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Mainstreaming New Measures 



by Julia C. Blixrud, Assistant Executive Director, External Relations, ARL 



W ithin the past four years, "new measures" 
has entered the lexicon of library 
performance and assessmenf liferafure. 

In fhe 1990s, escalafing economic pressures and 
demands for accounfabilify from governing boards, 
universify adminisfrafors, and governmenf agencies 
sfrongly encouraged ARL member libraries and 
many ofher libraries fo find new ways fo illusfrafe 
fhe confribufion fheir collections and services 
provided fo fheir constifuencies. Tradifional, inpuf 
measures (i.e., size of collecfions, budgef, and sfaff) 
were nof meefing insfifufional accounfabilify needs. 
Alfhough efforfs were in place fo measure some 
forms of oufpuf — e.g., activifies such as circulafion, 
inferlibrary borrowing, and group presenfafions — 
fhese, foo, did nof always provide a good or complefe 
indicafion of library performance and fhe cosf 
effectiveness of fheir services. In fhe 1990s, fhe ARL 
Sfafisfics and Measuremenf Commiffee began 
discussing fhe need fo move beyond fhe fradifional 
measures, building on earlier work in organizafional 
developmenf conducfed by ARL's Office of 
Managemenf Services in fhe lafe 1980s and 
fhroughouf fhe 1990s. ^ In early 1999, fhe ARL 
New Measures Inifiative officially began fo develop 
measures fhaf would better demonsfrafe libraries' 
insfifufional value and describe fheir currenf 
operations. 

Now after four years of infensive work, fhe New 
Measures Inifiafive is no longer new. ARL member 
libraries have infegrafed some of fhe measures info 
fhe mainsfream of library operafions and fhe 
Sfafisfics and Measuremenf program has an active 
agenda for research, experimenfafion, and 
demonsfrafion projecfs. 



Development of the Initiative 

ARL has been collecting descriptive dafa from ifs 
members for dozens of years and a robusf dafa file 
covers 1907-08 fo fhe presenf. However valuable 
fhese dafa are for describing long-esfablished 
characferisfics of research libraries and for 
monitoring frends, fhey are nof adequafe fo 
address emerging uses of technology for access fo 
information nor are fhey helpful for describing fhe 
changing nafure of service in research libraries. Over 
fhe years ARL experimented wifh gafhering dafa fo 
assess library performance. As early as 1982, ARL 
began a Supplemenfary Sfafisfics form fo fesf fhe 
gafhering of difterenf sfafisfics, and if fhe variables 
proved useful, fhey were moved info fhe main 
Sfafisfics. For example, collecting dafa on 
governmenf documenfs and reference fransacfions 
was fesfed in fhe Supplemenfary Sfafisfics and fhose 
items were added fo fhe main Sfafisfics in 1990-91 
and 1994-95 respectively. 

If was fhrough fhe Supplemenfary Sfafisfics fesf 
bed fhaf ARL began collecfing information on fhe 
characfer of research library invesfmenf in elecfronic 
resources. The definifional problems immediately 
became apparenf, as did fhe difficulties for libraries 
in readily collecfing some of fhe dafa. Befween 1996 
and 1998, Timofhy Jewell (Universify of Washingfon) 
served as an ARL Visiting Program Officer fo assess 
early efforfs fo collecf measures of elecfronic 
resources. The goal of his inquiry was fo develop 
sfandard definitions for more systematic collecfion 
of information abouf fhe fransformafion of research 
libraries. Through a series of meefings, discussions, 
reporfs, and analysis of existing dafa, fhe Jewell 
sfudy documented a communify seeking fo measure 
facefs of a complex environmenf subjecf fo radical 
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shifts due to rapidly changing technology and 
applications. Not surprisingly, his final report 
highlights the wide range of efforts underway to 
capture this information but also two concerns: that 
any information gathered have clear utility, and that 
time and effort required to supply the information 
be minimized. 

The tenor of library measurement discussions in 
the late 1990s was one of frustration peppered with 
examples of false starts in an environment that changed 
faster than it could be measured. There was a sense of 
urgency to develop better measures for the networked 
environment but there was 
no sense of commonality 
regarding which areas were 
most important to work on. 

What emerged from the 
experience of the late 1990s 
was a realization that any 
definition of new measures 
would need to be accepted 
by the community as part of the larger context of 
institutional assessment. There was also a realization 
that developing new measures carried costs and that 
this work had to be shared in order to minimize the 
expenses for any single institution.^ 

ARL Director Retreat 

Seeing the need for a common direction, Carla Stoffle 
(University of Arizona) and Paul Kobulnicky 
(University of Connecticut), chairs of the ARL Statistics 
and Measurement Committee and the Research Library 
Leadership and Management Committee, called a 
retreat in early 1999 in Tucson, Arizona. The goal of the 
retreat was to discuss how to develop strategies that 
would address the development of new measures.^ 

Retreat participants identified eight areas of interest 
for which measures would be particularly helpful: 



methodologies available to the full ARL membership 
as well as the broader library community. Retreat 
participants agreed that not all ARL members would be 
required to participate in all projects, which would give 
each project the flexibility it needed to test and refine 
measures without placing undue burdens on the entire 
ARL membership. In addition, it was proposed that 
these new projects would not be used to modify the 
existing data-collection efforts without considerable 
testing and that the traditional measures would be 
"frozen" in their current form in order to retain their 
integrity. As new measures became available, the ARL 

Statistics and Measurement 
Committee would review 
each one for addition to the 
regular ARL data- 
collection activities. 

After the retreat, brief 
working papers were 
developed by directors for 
some of the eight areas to 
expand on the ideas expressed and to identify possible 
approaches for investigation.^ Authors were 
encouraged to build upon previous research to better 
ensure early success for the New Measures Initiative. 
The areas of interest were tested with the broader 
ARL community at the May 1999 Membership Meeting 
and plans were set in motion to begin developing 
the measures. 

Progress in Areas of Interest 

The Statistics and Measurement Committee, under the 
leadership of Carla Stoffle, directed the New Measures 
Initiative. An ambitious agenda was set forth directing 
attention to a number of areas. A series of projects and 
special events were initiated in 2000 and considerable 
progress has been made in nearly all of the areas of 
interest identified at the retreat. 



What emerged from the experience of the late 
1990s was a realization that any definition of 
new measures would need to be accepted by the 
community as part of the larger context of 
institutional assessment. 




• User Satisfaction 

• Market Penetration 

• Ease and Breadth of Access 

• Library Impact on Teaching and Learning 

• Library Impact on Research 

• Cost Effectiveness of Library Operations 
and Services 

• Library Facilities and Space 

• Organizational Capacity 

In order for any specific new measures projects to 
succeed, it was determined that there must be 
collaboration among member leaders with strong 
interest in them; individual projects could and should 
be developed with different models for exploration; and 
there must be an intent to make resulting tools and 



User Satisfaction and Market Penetration 
Combining two of the retreat's areas of investigation — 
reaching the library community and identifying the 
success libraries have serving users — resulted in the 
largest new measures activity: LibQUAL-t-^''^.^ This 
research and development project undertaken by ARL 
in collaboration with Texas A&M University is defining 
and measuring library service quality across institutions 
and creating useful quality assessment tools. The Web- 
based, total-market survey measures library user 
perceptions of service quality and identifies gaps 
between minimum, desired, and perceived expectations 
of service. It provides project participants with reliable, 
tested, comparable data for internal and external 
benchmarking. In addition, a number of training 
activities have been developed to assist libraries in 
deploying the survey, interpreting results, and 
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implementing changes in their operations in response to 
the survey data. Colleen Cook and Fred Heath have led 
this effort from Texas A&M and Marfha Kyrillidou and 
Duane Websfer have led fhis efforf from ART. Two 
leading researchers from Texas A&M, Yvonna Lincoln 
and Bruce Thompson, have served as qualifafive and 
quanfifafive experfs respecfively and fhe projecf's 
developmenf has been well documenfed in fhe 
liferafure.® 

Two evenfs jumpsfarfed fhe LibQUAL+T*^ projecf. 
Firsf, ART held a symposium on measuring service 
qualify in Ocfober 2000.^ Experfs in fhe field of library 
service qualify and ART members parficipafing in a 
pilof projecf fo fesf a library user survey based on 
SERVQUAL — a service qualify evaluation fool used in 
fhe privafe secfor — spoke abouf fhe need for an 
insfrumenf fhaf looked af library service from fhe user's 
poinf of view. The symposium raised inferesf in fhe 
LibQUAL+T*^ projecf specifically and fhe issue of 
measuring library service qualify generally. Also in fhe 
fall of 2000, ARL and Texas A&M received a fhree-year 
granf from fhe U.S. Deparfmenf of Educafion Fund for 
fhe Improvemenf of Posfsecondary Educafion (FIPSE) 
fo fesf fhe LibQUAL+T*^ insfrumenf in a broad range 
of insfifufions. 

Begun wifh only 12 ARL libraries in 1999, fhe 
LibQUAL+T’'^ projecf has now reached fhe end of ifs 
FIPSE granf and is infernafional in scope wifh jusf over 
300 libraries parficipafing in fhe spring 2003 survey. 
Since fhe projecf began, over 250,000 individuals have 
responded fo fhe survey, creating fhe richesf dafasef of 
information on user perception of library service qualify 
fhaf exisfs. Alfhough mosf of fhe 2003 parficipanfs were 
academic libraries, specialized libraries such as healfh 
sciences and law also parficipafed. Consortia such as 
OhioLINK, fhe NY3Rs, and fhe Nefwork of Alabama 
Academic Libraries (NAAL) have provided financial 
supporf for fheir members and received reporfs fhaf 
analyze fhe survey dafa for consortia. For fhe firsf time 
in 2003, fhe profocol was applied fo a sef of public 
libraries in New York; a British English version was 
fesfed wifh Unifed Kingdom posfsecondary libraries, 
and a Canadian French franslafion of fhe insfrumenf 
was prepared fo supporf fhe parficipafion of French 
Canadians. Resulfs from fhis year's projecf are 
summarized in an article on page 8. Affer 2003, 
LibQUAL+T*^ will evolve info an ongoing service 
qualify assessmenf program af ARL. 

Addifional funding from fhe National Science 
Foundafion has been secured fo adapf fhe LibQUAL+T*^ 
insfrumenf for assessing digifal library service qualify 
wifhin fhe National Science Digifal Library (NSDL) 
framework. Qualifafive re-grounding is ongoing wifh 
sife visifs, inferviews, and focus groups held af fhe Mafh 



Forum, DLESE (Digifal Library for Earfh Sysfem 
Educafion), and MERLOT (Multimedia Educational 
Research for Learning and Online Teaching) fo dafe.^ 
Developing an undersfanding of fhe dimensions of 
digifal library service qualify is surfacing quesfions 
relafed fo elecfronic services, qualify, and educational 
impacf, an area where relatively little fheorefical and 
practical undersfanding exisfs fo dafe. A reporf on fhe 
evaluafion work of fhe NSDL appears on page 17. 

Ease and Breadth of Access 

The increasing use of elecfronic resources, documenfed 
by expendifures reporfed in ARL Supplemenfary 
Sfafisfics and confirmed by LibQUAL-i-T“ findings each 
year, was anofher area idenfified by refreaf parficipanfs 
as a high priorify for new measures. In February 2004, 
anofher sef of ARL members began fhe E-Mefrics 
projecf, an efforf fo explore fhe feasibilify of defining 
and collecting dafa on fhe use and value of elecfronic 
resources. ARL had some experience in fracking 
expendifures for elecfronic resources fhrough fhe ARL 
Supplemenfary Sfafisfics buf fhere was recognifion fhaf 
much more work was needed. The E-Mefrics projecf 
was coordinafed by Projecf Co-Chairs Sherrie Schmidf 
(Arizona Sfafe) and Rush Miller (Pittsburgh). ARL 
confracfed wifh Florida Sfafe Universify's Information 
Use Managemenf and Policy Insfifufe fo direcf a sfudy, 
gafher dafa, and do fhe necessary analysis. The projecf 
was conducfed in fhree phases: (1) an invenfory of 
whaf libraries were already doing in fhis area and 
idenfificafion of any libraries fhaf could provide besf 
practices; (2) idenfificafion and fesfing of dafa elemenfs 
fhaf could be collecfed and used as measures of 
elecfronic resources for frend analysis and 
benchmarking; and (3) analysis of fhe connection 
befween fhe use of elecfronic resources and insfifufional 
oufcomes. Using surveys, consulfafions, field fesfs, 
and sife visifs, fhe projecf invesfigafors idenfified 
16 measures in four cafegories (resources, use, 
expendifures, and digifizafion) along wifh fhree 
performance indicafors fhaf ARL libraries mighf use 
fo frack elecfronic resource use. E-Mefrics projecf 
documenfs can be found on fhe projecf's Web sife® 
and were also published in prinf. 

A relafed ARL Task Force on Sfafisfics from 
Vendor-Based Dafabase Producfs focused on working 
wifh vendors fo arrive af agreed-upon dafa elemenf 
definifions and ferms, specific dafa fhaf can be collecfed, 
and mefhods for reporting dafa fo libraries. This fask 
force builf on existing work in fhe field. ARL also 
parficipafed in infernafional efforfs culminating in early 
sponsorship of Projecf COUNTER (Counfing Online 
Usage of NeTworked Elecfronic Resources), an 
infernafional initiative designed fo serve librarians, 
publishers, and infermediaries by facilifafing fhe 
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recording and exchange of online usage statisticsd° 
Release 1 of fhe COUNTER Code of Practice, which 
focuses on journals and dafabases, was agreed fo and 
issued in December 2002. 

Affer completion of fhe E-Mefrics sfudy, ARL 
parficipafing libraries decided fo continue furfher 
invesfigafion and fesfed fhe proposed dafa elemenfs 
again in 2002. Gordon Frefwell (Universify of 
Massachusetts) served as a Visifing Program Officer 
during fhe firsf half of fhe year and Marfha Kyrillidou 
complefed fhe dafa collection in fhe latter half. This 
fesfing experience led fo fhe decision fo open fhe 
2003-04 pilof process fo addifional parficipanfs. The 
dafa elemenfs collecfed as parf of fhis projecf fhen will 
move info fhe regular ARL Supplemenfary Sfafisfics 
collection cycle for fhe year sfarfing in July 2004. Mosf 
of fhese ifems relafe fo fhe number and use of elecfronic 
resources and also seek fo describe emerging digifal 
library operafions. 

Library Impact on Teaching and Learning 
For many insfifufions, fhe accredifafion process has 
provided fhe sfrongesf impefus for a shifting campus 
emphasis on insfifufional oufcomes. The refreaf 
parficipanfs recognized fhaf some of fhe more difticulf 
measures fo develop would identify how a library 
confribufes fo fhe feaching and learning process. In 
order fo address fhis issue, several ARL members 
provided funds fo confracf wifh Eller Distinguished 
Service Professor Kennefh Smifh (Universify of 
Arizona) fo prepare a whife paper, "New Roles and 
Responsibilities for fhe Universify Library: Advancing 
Sfudenf Learning fhrough Oufcomes Assessmenf."^^ 
Smifh's paper encourages libraries fo develop sfrafegies 
for becoming involved in campus assessmenf acfivifies. 
He encouraged libraries fo: 

• Move from a confenf view (books, subjecf 
knowledge) fo a compefency view (whaf sfudenfs 
are able fo do) 

• Undersfand learning oufcomes of academic 
degree programs 

• Develop curriculum segmenfs or "offerings" 
fhrough which fhe library achieves learning 
oufcomes. 

ARL subsequenfly esfablished a Learning Oufcomes 
Working Group under fhe leadership of Carla Sfoftle fo 
expand on Smifh's recommendations and idenfify 
measures fhaf would describe fhe library's confribufion 
fo sfudenf learning oufcomes. Several fasks were puf 
forward fo fhe group: survey accredifafion agencies fo 
idenfify besf pracfice insfifufions, compile learning 
oufcomes used by ARL insfifufions for specific 
academic deparfmenfs or general education 
requiremenfs (if fhey exisf), confacf professional 



associations fo idenfify fhose fhaf are working in fhis 
area, idenfify specific offerings fhaf libraries could 
develop fo meef oufcomes fhus far idenfified, esfablish 
a plan fo fake fhe offerings fo faculfy in a selecf group of 
insfifufions, explore wifh ACRL fhe offering of fraining 
on informafion liferacy skills, define and creafe 
generalized fools for assessmenf, and, perhaps mosf 
imporfanfly, define whaf is meanf by learning oufcomes 
and af whaf level ARL should focus ifs affenfion 
(sfudenf, course, program, deparfmenf, efc.). 

The working group began by scanning fhe higher 
education environmenf for consensus on or besf 
practices in idenfifying learning oufcomes, wifhouf 
much success. The resulfs confirmed fhe imporfance, 
buf also fhe difticulfies, of advancing fheir agenda: 

• Nafional organizafions had nof yef agreed on a 
sfandard definifion of learning oufcomes. 

• Accredifing agencies were leaving fhe 
discussion of learning oufcomes up fo fhe 
individual campuses. 

• Since definifion of learning oufcomes is vague, 
mosf campuses have nof yef grappled wifh fhe 
issue — fhose fhaf have are doing if because of fhe 
pressure of accredifafion. 

• Idenfiflcafion of fhe people wifhin an 
organization concerned wifh learning 
oufcomes is difticulf. 

• Compilation of a common lisf of learning 
oufcomes from ARL members is nof yef possible. 

• Besf practices have been limifed fo smaller 
insfifufions — none are of fhe size and complexify 
of large research organizafions. 

• Faculfy feel fhreafened by campus discussions 
of learning oufcomes. 

Forfunafely, wifhin fhe pasf couple of years, some 
of fhese challenges are being addressed. Recenf 
liferafure on higher educafion assessmenf illusfrafes 
evidence fhaf general educafion oufcomes and, in some 
cases, oufcomes for some professional programs are 
being developed and used. Common fhemes and 
learning objecfives have arisen fhaf can be applied by 
insfifufions beginning fo develop fheir own oufcomes 
assessmenf eftorfs. Accredifing agencies are a key 
source in encouraging fhese developmenfs. The 
working group is preparing a whife paper fhaf 
summarizes fhe liferafure on higher educafion 
assessmenf, examines fhe definifions and sources of 
learning oufcomes, idenfifies domains of learning, 
describes fhe levels of assessmenf, and suggesfs 
frameworks for puffing fhese pieces fogefher wifhin 
an insfifufion. See fhe relafed arficle on page 14 of 
fhis issue for a summary of fhe whife paper. 
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One specific recommendation to discover libraries' 
contributions to the teaching and learning process is to 
begin with a process over which libraries have control, 
such as instruction, information literacy, or reference. 
ARL has taken that approach and is now partnering 
with Kent State University on Project SAILS 
(Standardized Assessment of Information Literacy 
Skills), which has received partial funding from the 
Institute of Museum and Library Services (IMLS) to 
develop an instrument for programmatic-level 
assessment of information literacy skills. SAILS is based 
on outcomes defined by the ACRL Information Literacy 
Competency Standards for Higher EducationP Seven 
libraries participated in Phase I of SAILS and met at the 
ALA Annual Conference in June 2003 to review their 
test results, suggest how data could be best formatted 
for use by libraries, and share their experiences in 
administering the survey to different student groups. 
Over forty libraries are participating in Phase II of the 
project, which will continue to test both the items in the 
survey as well as how best to administer the instrument 
within different library and academic settings. A fuller 
description of the SAILS Project appears on page 18. 

Library Impact on Research 
In 2001, Douglas Jones (University of Arizona) was 
appointed ARL Visiting Program Officer to identify 
issues and explore possible measures that demonstrate 
the role of the library in support of the university 
research mission. He investigated a set of dimensions 
that might lead to library success in support of the 
research mission of the university. His focus was on 
correlating research outcomes with different types of 
library support, identification of best practices at those 
institutions for which comparable levels of library 
support resulted in greater productivity, and interviews 
with key faculty and administrators to gather insights 
into how libraries might improve support for research. 
Some of the sources for institutional rankings and 
performance measures that might be used as indicators 
for research outcomes included U.S. News and World 
Report rankings. The Center (University of Florida) 

Top American Research Universities, the Association 
of American Universities Data Exchange (AAUDE) 

Data Warehousing Initiative, and the Integrated 
Postsecondary Education Data System (IPEDS).^'^ In a 
presentation to the ARL survey coordinators in January 
2002, Jones identified some possible research-related 
outcome measures: (1) scale of research (e.g., total R&D 
expenditures and federally sponsored R&D); (2) faculty 
(e.g., membership in national academies or other 
significant faculty awards indicating faculty 
distinction); (3) students (e.g., Ph.D.s awarded, post- 
doctoral appointments); (4) scholarly output (e.g., 
articles, books). Jones noted that it is difficult to isolate 



the library's contribution to research quality or 
productivity. There may be correlation between 
research outcome and library support, but causation is 
more difficult to prove. He suggested that libraries 
identify their institution's most productive researchers 
and ensure that their information needs are met. 

If researcher access to information in a networked 
environment contributes to their success, finding a way 
to capture data on use of resources is one way that a 
library can document its contributions. Brinley Franklin 
(University of Cormecticut) and Terry Plum (Simmons 
Graduate School of Library and Information Sciences) 
conducted patron use surveys of networked electronic 
services at five geographically disparate academic 
health science libraries and two main libraries between 
1999 and 2003. They developed a pop-up Web-survey 
methodology that can be exploited to identify not only 
how library patrons use electronic services, but also to 
what purpose that use was being made. This study, 
called Measuring the Impact of Networked Electronic 
Services (MINES), is becoming an ARL new measures 
project. See the article on page 20. 

Cost Effectiveness of Library Operations 
and Services 

Libraries continually seek ways to improve the cost- 
effectiveness of their operations. Retreat participants 
identified this area as one that needed to be addressed, 
but did not identify any specific operations to be 
explored. There was general agreement that the 1997 
ILL/DD Performance Measures study resulted in 
significant changes in the operations of interlibrary loan 
departments across the membership. Members of the 
Statistics and Measurement Committee were interested 
in a follow-up study to see whether departments had 
become more efficient and whether costs had been 
reduced or contained. A new study. Assessing ILL/DD 
Services, was begun in fall 2002. This self-funded study 
updated, replicated, and expanded the 1997 ARL 
ILL/DD Performance Measures Study and obtained 
data on the performance of mediated and user-initiated 
(unmediated) interlibrary loan/ document delivery 
operations in research, academic, and special libraries. 

A total of 75 ARL and non- ARL libraries participated. 
Tom Delaney (Colorado State University) and Bruce 
Kingma (Syracuse University) collaborated with Mary 
Jackson (ARL) in the study. See page 21 for an article on 
this project. ARL will publish a final report and sponsor 
a series of workshops to disseminate the findings. 

The Statistics and Measurement program also 
investigated the Technical Services Cost Study 
conducted by five universities across the country. 

Using a time- and cost-recording process developed by 
Dilys Morris (Chronos Group, formerly of Iowa State 
University), staff time on tasks performed within 
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different cost centers is tracked over several sampled 
time periods. A variety of work processes can be traced 
and the resulting data can be analyzed by employee 
type, time, non-staff costs (such as vendors, systems, 
utilities), etc. Software developed by this project will 
enable libraries to input their own data and use it for 
benchmarking against the collective results. 

Along the same lines, Eileen Hitchingham (Virginia 
Tech), with other members of the Statistics and 
Measurement Committee, worked on a Staff Allocation 
Project to determine if a set of spreadsheets could be 
developed into a tool for library managers to determine 
how staff time was being distributed across a range of 
functions. Two institutions, 

Notre Dame and Vanderbilt, 
will be combining the 
methodologies from the 
Technical Services Cost 
Study and the Staff Allocation Project to look at staff 
allocation costs across the library. See the article by 
Dilys Morris on page 23. 

Library Facilities and Space 
While no specific projects have begun yet regarding 
measurements of library space, this area has not 
languished. ART and OCLC cosponsored a Strategic 
Issues Forum, "Future Library Architecture: 
Conception, Design, and Use of Library Space" in 
February 2002 in Las Vegas, Nevada. The goals of the 
forum were (1) to engage library leaders in an 
exploration of important issues affecting the future 
development of library space and (2) to help prepare 
them to envision and make the case for innovative 
library spaces that are directly responsive to user needs 
and interests on campus. In addition, ARL's May 2003 
Membership Meeting examined the connections 
between the concepts of research libraries as place and 
as space and how they contribute to a sense of 
community.^® 

One question retreat participants asked about 
space was, "How can libraries tell if they are effectively 
utilizing their space?" Cliff Haka and Jim Hensley 
(Michigan State University), in an article on page 24, 
describe Michigan State University's effort to conduct a 
space-utilization study using CAD 2000 software. They 
postulate that this might be a useful tool for other 
libraries and could perhaps provide an opportunity for 
benchmarking the use of space in ARL libraries. 

Organizational Capacity 
Retreat participants acknowledged that one of the 
challenges facing research libraries is the importance of 
developing an institutional culture that understands the 
need to focus on new measures (i.e., outcomes) rather 
than solely on traditional measures. One way to change 
culture is to change what is measured. This shift in 



thinking requires library staff to not only learn new tools, 
but also to recognize the importance of data in making 
decisions that will improve library services and operations. 
ARL and the Office of Leadership and Management 
Services have developed a number of activities to support 
the learning needed by library staff. The LibQUAL-t-^*^ 
program has a range of training, including introductory 
workshops, project-planning sessions, data-review 
sessions, and post-survey seminars. The program also 
offers the Service Quality Evaluation Academy, an 
intensive five-day workshop focusing on qualitative and 
quantitative methods for collecting and analyzing library 
service quality data. In collaboration with the ARL/ OLMS 

Online Lyceum, a Measuring 
Library Service Quality 
Course has been offered to all 
LibQUAL-i-T*^ participants and 
a range of libraries interested 
in the background and theory of measuring service quality 
and methods to assess and improve service. The 
ARL /OLMS offers a two-day workshop on Creating a 
Culture of Assessment, which identifies the main forces 
influencing current library goals and work environments 
and how these forces demand the adoption of a new 
paradigm of thinking, new attitudes, and the support for 
structures that make assessment and evaluation an integral 
part of library work. See page 31 for more information. 

"Toward Rethinking Academic Library Performance 
in the Digital Age," an ARL/QCLC Strategic Issues Forum 
held in 2001, introduced senior library administrators to 
issues regarding performance measurement, and provided 
an overview of the Balanced Scorecard methodology 
as one means for libraries to establish performance 
management systems. The University of Virginia is one 
of the first libraries to use this methodology; an article by 
Jim Self describing their experience is on page 28. 

In 1998, the University of Arizona Library 
implemented a performance measurement system to 
assess itself. An article on page 26 describes Arizona's 
experience with the Performance Effectiveness 
Management System that aligns individual, unit, and 
whole organization efforts with library strategic goals and 
provides measures that indicate success, progress, and the 
need for improvement. The article also gives examples of 
benchmarks that Arizona identified in order to provide 
other libraries with a point of departure for measuring 
their own operations. 

Susan Beck (Rutgers University) served as an ARL 
Visiting Program Qfficer to investigate the impact of 
assessment on library management decision making and 
the degree to which assessment data has influenced 
organizational change. She studied nine public ARL 
institutions, selected primarily because they are involved 
in assessment activities. See the article on page 30 for a 
summary of her research. 



In order to change the culture, 
change what you measure. 
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Moving Measures into the Mainstream 

What does it take for an institution to move into a culture 
of assessmenf? Based on experience and research over 
fhe lasf few years, fhe following have emerged as 
essenfial elemenfs: 

• Resources (i.e., time and money) 

• Individual and insfifufional buy-in 

• Access fo individuals fo evaluafe 

• Expertise fo conducf evaluation 

• Projecf managemenf experience 

• Appropriafe benchmarks 

• Concepfual clarify 

• Measuremenf and design requiremenfs 

• Insfrumenf validify and reliabilify 

The ART New Measures Initiafive has worked fo 
assure fhaf all of fhese componenfs are parf of fhe 
projecfs underfaken. Affer several years of hard work 
festing and refining measures and developing a broader 
base of experience and expertise among sfaff in libraries, 
if is now fime fo begin moving many of fhe "new" 
measures info fhe mainsfream of library operafions. 

Moving new measures info fhe mainsfream of fhe 
ART Sfafisfics and Measuremenf program does nof mean 
ART is fhrowing fhe baby ouf wifh fhe bafhwafer. The 
dafa ART fradifionally gafhers are sfill imporfanf for 
documenting fhe fransformafion of research library 
collections, services, and sfaff, and for idenfifying frends 
in fhe cosfs of operafions. However, fhe changing nafure 
of research libraries also calls for new and differenf 
measures fhaf nof only describe fhe library, buf also 
indicafe how well a library is serving ifs communify. 

There is no doubf fhaf demands for library 
accounfabilify will continue. The users of library services 
have an insafiable appefife for qualify informafion 
anyfime, anywhere. Parenf insfifufions are pressured fo 
demonsfrafe fheir value fo legislafors, funding 
aufhorities, sfudenfs, efc., and are asking for dafa from all 
organizational unifs, including libraries, in order fo 
provide if. An exploding growfh in fhe use and 
applicafions of fechnology along wifh an increased 
competifion of resources makes if more imporfanf fhan 
ever for libraries fo use hard dafa fo make fhe case for 
resources. This means libraries will have fo confinually 
measure and monitor fheir effectiveness and, as 
fechnology and user behavior changes, sfrive fo develop 
measures fhaf have fhe greafesf potential in helping 
library managers lead change. 

The Success of the New Measures Initiative 

There are several characferisfics of success of fhe ART 
New Measures Inifiafive. The firsf has been fhe 
commifmenf by individual insfifufions fo fake leadership 



for specific projecfs. As highlighfed in fhis article, 
individual insfifufions and library directors have faken 
leadership for new measures projecfs. Their personal 
inferesf and dedication fo fhe inifiafive ifself or fo a 
parficular area for invesfigafion carmof be oversfafed. 
They have ensured nof only fhaf projecfs are well 
defined and launched, buf also have kepf fhe 
enfhusiasm for fhe work in fhe torefronf of fhe ART 
communify. Anofher characferisfic has been fhe projecf 
nafure of fhe inifiafive. As agreed fo by refreaf 
parficipanfs, if was evidenf fhaf no single measure or sef 
of measures would work fo address all eighf areas of 
concern. By allowing a variefy of projecfs and 
approaches, fhe inifiafive was able fo quickly address all 
of fhe areas relafive fo fhe degree of inferesf expressed 
by individual projecf parficipanfs. Also, member 
libraries were nof obligafed fo join all projecfs. They 
could choose only fhose fhaf were of mosf inferesf based 
on local sifuafions and fhose choices guaranfeed fheir 
inferesf and acfive parficipafion. Joining a projecf could 
mean projecf leadership, acfive parficipafion by fhe 
direcfor and sfaff, or supporf fhrough funding. As 
resulfs of each of fhe projecf are made available, fhe full 
ART communify as well as fhe larger library 
communify benefifs. Anofher aspecf of success for fhese 
projecfs is fhe opening up of some of fhem fo fhe wider 
library communify. The cross-ferfilizafion especially 
wifh ofher academic insfifufions has allowed fhe 
projecfs fo expand fheir supporf base and sfrengfhened 
fhe accepfance of fhe fools and mefhodologies 
fhroughouf fhe library communify. 

The dafa fhaf libraries collected and reported in fhe 
pasf were primarily inpuf dafa (how much do you 
have). In fhe 1990s fhere was encouragemenf fo provide 
more oufpuf dafa (how much do you do). Libraries are 
now in an environmenf fhaf asks for oufcomes (whaf is 
fhe benefif of your service) or impacf (whaf difference 
do you make). Insfifufions are idenfifying fheir 
oufcomes, defining criferia for success, and determining 
mefhods fo look for evidence of success. ART has 
demonsfrafed fhaf if is possible fo move from inpuf fo 
impacf measures fhrough ifs New Measures Inifiafive. 
As fhe inifiafive becomes a regular componenf of fhe 
Sfafisfics and Measuremenf program, ifs success factors 
can serve as elemenfs in a model for ofher 
organizafional unifs fo use as fhey look fo improve fheir 
own insfifufional effectiveness. 

The year 2003 is fransifional for fhe New Measures 
Inifiafive — leadership of fhe Sfafisfics and Measuremenf 
Committee passed from Carla Stoffle fo Brinley Franklin. 
The inifiafive has mafured and been infegrafed info fhe 
Sfafisfics and Measuremenf program. For example, ART 
has assumed full responsibilify for LibQUAL-i-T*^, plans 
are in place for E-Mefrics fo move info fhe ARE 
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Supplementary Statistics, Project COUNTER is being 
implemented, and results of the new ILL/DD study will 
be used by libraries to improve their operations. We 
have reached a point where some measures that have 
been successfully fesfed will be moved fo fhe main dafa- 
collecfion program while ofhers are dropped from ARL's 
surveys. Ofher measures will continue fo be fesfed 
among a larger communify in a continuous cycle of 
evaluafe and deploy or drop, ideas for new measures 
continue fo be welcomed by fhe program. Several years 
ago our challenge was fo develop our own new 
measures or fhey would be defermined by ofhers 
for us. We have mef fhaf challenge successfully 
fhrough fhe New Measures Initiative. 

1 Sarah Pritchard, "New Directions for ARL Statistics," ARL 
Bimonthly Report, no. 161; Martha Kyrillidou and William Crowe, 

"In Search of New Measures," ARL Bimonthly Report, no. 197 (April 
1998), <http: / / www.arl.org/newsltr/ 197 / newmeas.html>. 

2 ARL characterized some of the issues surrounding measurement in 
research libraries in a special issue of its newsletter, "Issues in 
Research Library Measurement," ARL Bimonthly Report, no. 197 
(April 1998), <http: / / www.arl.org/newsltr/197 / 197toc.html>. 

^ For those readers not familiar with the development of the New 
Measures Initiative, it was introduced in the December 1999 ARL 
Bimonthly Report, no. 207, <http: / / www.arl.org / newsltr / 207 / >. 

^ <http: / / www.arl.org / stats / newmeas / nmbackground.html>. 

^ <http:/ / www.libqual.org/>. 

® <http:/ / www.libqual.org/Publications/>. 

^ "Measuring Service Quality," Martha Kyrillidou and Fred M. Heath, 
eds. Library Trends 49, no. 4: 541-799, <http:/ / www.arl.org/ 
libqual / events / oct2000msq/ program.html>. 

® <http: / / mathforum.org/>, <http: / / www.dlese.org/dds/>, 

<http: / / www.merlot.org / Home.po>. 

® <http: / / www.arl.org / stats / newmeas / emetrics / >. 

<http: / / www.projectcounter.org / >. 

<http: / / www.arl.org / stats / newmeas / outcomes / 
HEOSmith.html>. 

12 Examples of how research libraries are engaged in developing 
curriculum segments and offering instructional technology services 
are available in the responses to the 2002 ARL Survey on New 
Approaches to Collections and Access services, 

<http:/ / db.arl.org /Casort/>. 

12 Information Literacy Competency Standards for Higher Education, 
approved by the Association of College and Research Libraries in 
January 2000, <http: / / www.ala.org/Content/NavigationMenu / 
ACRL / Standards_and_Guidelines/Information_Literacy_Compete 
ncy_Standards_for_Higher_Education.htm>. 

1^ <http: / / www.usnews.com / usnews / edu / college / rankings / 
rankindex_brief .php>, <http: / / thecenter.ufl.edu / >, 

<http: / / www.pb.uillinois.edu / AAUDE / >, 

<http: / / nces.ed.gov / ipeds / >. 

12 Mary Jackson, Measuring the Performance of Interlibrary Loan 
Operations in North American Research & College Libraries 
(Washington, D.C.: ARL, 1998). 

12 <http: / / www.arl.org / arl / proceedings / 142 / >. 



Measuring User 
Satisfaction and 
Market Penetration 



Library Users Assess Service 
Quality with LibQUAL+tm 
AND E-QUAL 

by Kaylyn Hipps, Editorial and Research Associate, ARL, 
and Martha Kyrillidou, Senior Program Officer for Statistics 
and Measurement, ARL 

L ibQUAL-i-i'M is a suife of services aimed af 
measuring library user percepfions and 
expecfafions of service qualify. Af fhe core of fhe 
projecf is a Web-based survey fhaf seeks feedback from 
library users abouf fheir experiences wifh fhe library 
and fhe exfenf fo which fhe library meefs fheir needs. 
The survey was developed by a feam from ARL and 
Texas A&M Universify fhrough a combination of 
qualifafive inferviews wifh Norfh American library 
users and four annual survey iferafions fo dafe, sfarfing 
in 2000. The U.S. Deparfmenf of Educafion's Fund for 
fhe Improvemenf of Posfsecondary Educafion (FIPSE) 
provided parfial supporf for fhe firsf fhree years of 
fhe projecf. 

Affer four iferafions and revisions of fhe survey 
insfrumenf, fhree dimensions of library service qualify 
have surfaced: 

• library as place (ufilifarian space, symbol, refuge); 

• information confrol (scope, fimeliness, 
convenience, ease of navigafion, modern 
equipmenf); and 

• affecf of service (empafhy, responsiveness, 
assurance, reliabilify). 

In fhe spring of 2003, more fhan 120,000 library 
users from over 300 libraries responded fo fhe 
LibQUAL-i-i'M survey. These numbers include over 
25,000 users af 55 ARL academic libraries, who said, in 
fhe aggregafe, fhaf fheir libraries were mosf successful 
in providing physical facilities fhaf meef fheir needs and 
in providing skilled and caring sfaff. These users 
expressed fhe highesf expecfafions for fhose services 
and fools fhaf enable fhem fo easily gain access fo and 
use informafion independenfly; fhese expecfafions are 
parf of fhe library service qualify dimension called 
"informafion confrol." The area fhaf ARL academic 
library users idenfified as especially needing 
improvemenf is access fo informafion fhaf fhey need 
for fheir work, such as journal collections and ofher 
maferials in bofh prinf and elecfronic form. This is 
also parf of fhe informafion confrol dimension. 
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Participating institutions are using their 
LibQUAL+T’'^ survey data to implement change 
within their libraries. Many of the libraries already 
knew that some changes were necessary; the survey 
confirms fhose beliefs and provides evidence fo jusfify 
fargefed funding increases. Some libraries are using 
additional dafa-gafhering mefhods — such as focus 
groups and local surveys — fo supplemenf fheir 
LibQUAL+T’“ resulfs or invesfigafe fhem in more 
defail. If is also common for libraries who have 
parficipafed in LibQUAL+T*^ fo sfrengfhen fheir 
communicafions wifh users in order fo better inform 
fhem and manage fheir expecfafions. Libraries find 
fhe user commenfs fhaf are gafhered as parf of fhe 
survey exfremely helpful as fhey inferpref fhe resulfs 
from fhe sfandardized parf of fhe survey in fhe confexf 
of fheir local environmenfs.^ 

The LibQUAL+T’'^ dafa sef is by far fhe largesf 
of ifs kind. Each armual survey iferafion has seen 
fremendous growfh in fhe numbers of parficipafing 
libraries and survey respondenfs as well as a 
broadening of fhe fypes of libraries. The number of 
parficipafing insfifufions increased from 12 in 2000 fo 
308 in 2003. The 2003 participanfs included college and 
universify libraries, communify college libraries, healfh 
sciences libraries, milifary libraries, public libraries, and 
sfafe libraries. The survey was complefed by more 
fhan 5,000 users in 2000 and more fhan 120,000 in 2003. 
Libraries in four counfries — fhe U.S., Canada, fhe U.K., 
and fhe Nefherlands — parficipafed in fhe 2003 survey. 

For fhe firsf time, in 2003, fhe LibQUAL+T*^ survey 
was franslafed info languages ofher fhan American 
English: a Brifish English version was adminisfered in 
20 U.K. insfifufions, a Dufch English version in one 
insfifufion in fhe Nefherlands, and a Canadian French 
version in fwo universifies in Canada. An analysis of 
fhe survey dafa from fhe French Canadian insfifufions 
showed fhaf fhe franslafion process produced scores 
fhaf are equivalenf fo fhe English versions of fhe 
insfrumenf. The LibQUAL+T’*’ feam is confidenf 
fhaf fhe differenf versions of fhe survey are 
"culfurally relevanf" in fheir respecfive languages 
and "concepfually equivalenf fo fhe original" 

American English survey.^ 

LibQUAL+T’*’ is now offered as an ongoing ARL 
program fo individual libraries of all fypes. In 
addition, many libraries have parficipafed in fhe 
program as parf of consorfia. Regisfrafion for fhe 
spring 2004 survey is underway fhis fall. 

e-QUAL 

The mosf recenf developmenf in fhe LibQUAL+^’*’ 
projecf is fhe feam's adapfafion of fhe survey fo fhe 
digifal library environmenf. This efforf, called 
e-QUAL, is parfially supporfed by a granf from fhe 



Nafional Science Foundafion, Nafional Science Digifal 
Library (NSF/NSDL). The overarching goal of 
e-QUAL is fo develop a digifal library service qualify 
assessmenf process fhaf enhances sfudenf learning by 
permitting fhe allocation of resources fo areas of 
user-identified need. The e-QUAL projecf seeks fo 
accomplish fhis goal by achieving fhe following 
objectives and oufcomes: 

• define fhe dimensions of digifal library service 
qualify from fhe perspective of fhe users; 

• develop a fool for measuring user perceptions 
and expecfafions of digifal library service 
qualify across NSDL digifal library confexfs; 

• idenfify digifal library "besf pracfices" fhaf 
permif generalizations across operations and 
developmenf platforms; 

• enhance sfudenf learning by effectively 
managing user percepfions and expecfafions of 
digifal library services; 

• esfablish a digifal library service qualify 
assessmenf program as an infegral parf of fhe 
library service qualify assessmenf program af 
ARL; and 

• insfifufionalize continuous producf and process 
evaluafion efforfs direcfed toward positive and 
timely managemenf of oufcomes.^ 

Many digifal libraries are sfill in fheir formative 
sfages of developmenf so fhe creation of an evaluafion 
fool like fhe envisioned e-QUAL survey needs fo be 
underfaken wifh care. To avoid premafure evaluafion 
and misguided comparisons across very differenf 
digifal libraries, fhe e-QUAL projecf advocafes using 
mixed mefhods fo develop a digifal library assessmenf 
fool and anticipates fhaf e-QUAL evenfually will be 
one of several such fools for digifal libraries. 

The e-QUAL projecf is currenfly in fhe qualifafive 
sfage of research. The feam is sfudying digifal 
libraries — such as fhe Mafh Forum, MERLOT, and 
DLESE — via inferviews, site visifs, and focus groups. 
The qualifafive analysis will help fhe projecf 
undersfand digifal library users' percepfions of 
service qualify. 

Alfhough fhere are differenf working models of 
digifal libraries — e.g., some confain acfual collecfed 
maferials, posfed online; ofhers confain links fo 
maferials hosfed elsewhere; ofhers are developing 
infrasfrucfure fools — fhey do share fhe goal of 
connecting users wifh information via fhe Infernef. 
Characferisfics of digifal libraries fhaf may be 
evaluafed include information access, sense of 
communify, and qualify of maferials. 




^ a 
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The e-QUAL project will develop and test an initial 
set of survey questions in late 2003. A second round of 
fesfing is planned for spring 2004. 

The LibQUAL+T“ and e-QUAL feam members 
include: 

Consuella Askew, LibQUAL-t-^^ Program Specialisf, 
ARL 

Julia Blixrud, Assisfanf Execufive Director, External 
Relafions, ARL 

Colleen Cook, Inferim Dean of Universify Libraries 
and Professor and Wrighf Chair of Library 
Science, Texas A&M Universify 

Fred Heafh, Vice Provosf and Director of General 
Libraries, Universify of Texas af Ausfin 

Amy Hosefh, New Measures Projecf Assisfanf, ARL 

Marfha Kyrillidou, Senior Program Officer for 
Sfafisfics and Measuremenf, ARL 

Yvonna Lincoln, Distinguished Professor and Rufh 
Harrington Chair of Education, Educational 
Adminisfrafion, Texas A&M Universify 

Jonafhan Sousa, Technical Applications Developmenf 
Manager for New Measures, ARL 

Bruce Thompson, Professor and Distinguished 
Research Scholar, Educational Psychology, 

Texas A&M Universify 

Duane Webster, Execufive Direcfor, ARL 

Mark Young, Research Assisfanf for Sfafisfics 
and Measuremenf, ARL 

For more information about LibQUAL+™, see 
<http://www.libqual.org/>. For more about e-QUAL, 
<http://www.arl.org/stats/newmeas/emetrics/nsdl.html>. 

1 For more on how libraries use LibQUAL+’'“ data, see Consuella 
Askew Waller and Kaylyn Hipps, "Using LibQUAL+’’“ and 
Developing a Culture of Assessment in Libraries," ARL Bimonthly 
Report, no. 221 (April 2002): 10-11, <http: / / www.arl.org/ newsltr/ 
221 / libqarl.html>; and Robyn Huff-Eibl and Shelley Phipps, "Using 
LibQUAL+’'“ Results at the University of Arizona: Responding to 
Customer Input — Listening and Acting," ARL Bimonthly Report, no. 
221 (April 2002): 12-13, <http:/ / www.arl.org/newsltr/221/ 
libqaz.html>. 

2 Martha Kyrillidou, Toni Olshen, Fred Heath, Claude Bonnelly, and 
Jean-Pierre Cote, "Cross-Cultural Implementation of LibQUAL+’'“: 
The French Language Experience," presented at the Fifth 
Northumbria International Conference on Performance 
Measurement in Libraries, Durham, England, July 28, 2003. 

^ Colleen Cook, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, 
Bruce Thompson, and Duane Webster, "Developing a National 
Science Digital Library (NSDL) LibQUAL-i-’’“ Protocol: An 
E-Service for Assessing the Library of the 21st Century," presented 
at the NSDL Evaluation workshop, Washington, D.C., October 
2003, <http:/ /www.libqual.org/documents/admin/NSDL_ 
workshop_web.pdf>. 



A LibQUAL+tm Primer 

Excerpted from Colleen Cook, Fred Heath, and Bruce 
Thompson, "LibQUAL+™ from the U.K. Perspective," 
presented at the Fifth Northumbria International Conference 
on Performance Measurement in Libraries, Durham, England, 
July 28, 2003, <http://www.libqual.org/documents/admin/ 
nor thumb _CC.pdf>. 

L ibQUAL-i-T’'^ is only one of 11 difterenf ways to listen 
to customers, called a "fofal markef survey."^ Berry 
recommended using multiple listening mefhods, 
and emphasized fhaf "Ongoing dafa collection. . .is a 
necessify. Transactional survey, fofal markef surveys, 
and employee research should always be included." 

LibQUAL-i-T''^ was modeled on fhe 22-ifem 
SERVQUAL fool developed by Parasuraman, Berry and 
Zeifhaml.^ ...However, SERVQUAL has been shown to 
measure some issues nof particularly relevanf in 
libraries..., and to nof measure some issues of 
considerable inferesf to library users. 

The final 22 LibQUAL-t-^''^ ifems were developed 
fhrough several iferafions of quanfifafive sfudies involv- 
ing a larger pool of 56 ifems. These 56 ifems were 
fhemselves identified following qualifafive research 
inferviews wifh library sfudenf and faculfy users af 
several difterenf universifies. 

In conducfing service qualify assessmenfs, if is 
absolutely critical fo demonsfrafe fhaf fhe assessmenf 
scores measure somefhing (as againsf nofhing). Scores 
measure nofhing when fhey are random. In psychomef- 
rics, fhe properfy of dafa measuring somefhing is called 
"reliabilify." In previous adminisfrafions, LibQUAL-t-^''^ 
scores have been repeafedly shown fo be reliable. 

Even when assessmenf scores are demonsfrafed fo 
be reliable, if additionally musf be shown fhaf fhe scores 
measure fhe intended consfrucfs. [This] is called 
"validify." A primary fool in such proofs involves fhe 
sfafisfical mefhod called factor analysis. Various 
mefhods have been employed in prior sfudies 
suggesting fhaf LibQUAL-t-^''^ scores are valid. 

There are fwo primary ways fo inferpref 
LibQUAL-i-T’^. Firsf, scores on perceptions may be 
compared againsf scores on whaf is reported fo be mini- 
mally accepfable service, and whaf is reporfed fo be 
desired service; fhis is called fhe "zones of folerance" 
inferprefafion framework. 

Second, sfafisfical norms may be used fo characfer- 
ize facfually whaf percenfage of users or of insfifufions 
generated lower perception ratings. 

— Copyrighf 2003 Colleen Cook, Fred Heafh, and 
Bruce Thompson 

^ L.L. Berry, On Great Service: A Framework for Action (New York: 

The Free Press, 1995). 

2 A. Parasuraman, L.L. Berry, and V.A. Zeithaml, "Refinement and 
Reassessment of the SERVQUAL Scale," Journal of Retailing 67 
(1991): 420-450. 
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Measuring Ease 
AND Breadth of Access 



E-Metrics: Next Steps eor 
Measuring Electronic Resources 

by Julia C. Blixrud, Assistant Executive Director, External 
Relations, ARE, and Martha Kyrillidou, Senior Program 
Officer for Statistics and Measurement, ARE 

R esponding to user demand, libraries have steadily 
been shifting the focus of fheir collection 
developmenf fo fhe acquisition and licensing of 
elecfronic confenf, much of if via consorfia. In fhe lasf 
decade, fhe average percenfage of acquisition dollars fhaf 
ARE member libraries direcfed fo elecfronic resources rose 
from 3% fo 20%. In 2002, 110 ARE universify libraries 
reporfed spending more fhan $171 million on elecfronic 
resources, and 48 ARE libraries reporfed anofher $20 
million expended on fheir behalf fhrough cenfrally funded 
consortia.^ In spife of considerable efforfs, aggregafe dafa 
documenting fhe quanfify and use of fhese e-resources 
was proving fo be elusive. The ARE E-Mefrics work was 
esfablished in 2000 fo develop sfandard definitions for 
measures fhaf libraries could use fo describe: (a) fhe 
e-resources fhey make accessible, (b) fhe use made of fhe 
resources, and (c) fhe level of library expendifures. 

Background 

ARE began collecting dafa on elecfronic resources fhrough 
ifs Supplemenfary Sfafisfics survey in 1992-93.^ Many 
libraries were able fo reporf some rnifial cosf dafa buf 
affer several affempfs fhe libraries were unable fo counf 
consisfenfly fhe resources purchased or fhe use made of 
fhose resources. In 1996-98, Tim Jewell (Universify of 
Washingfon) served as an ARE Visiting Program Officer 
fo analyze fhe Supplemenfary Sfafisfics survey dafa on 
elecfronic resources from 1992-93 fo 1995-96. His projecf 
confirmed fhaf reporfed figures underesfimafed fhe acfual 
expendifures by an unknown amounf and fhaf local 
record-keeping practices made if difficulf fo respond fo 
some quesfions in fhe survey. The 1996-97 ARE 
Supplemenfary Sfafisfics quesfionnaire was revised fo 
fry fo capfure figures fhaf had been elusive in previous 
surveys. Jewell identified several ofher frends in fhe dafa 
fhaf member libraries were reporting and confirmed fhaf 
existing dafa was neifher comprehensive nor comparable.^ 
Recognizing fhe pace af which e-resource spending 
was increasing, if was clear fhaf consisfenf and reliable 
dafa was needed fo identify whefher fhe significanf 
invesfmenf in fhese resources was benefiting library 
users. In 1999, fhe parficipanfs af fhe ARE New Measures 
refreaf also identified elecfronic resources as an area of 
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continuing concern and in February 2000 a group of ARE 
direcfors and senior sfaff mef for fwo days fo defermine 
how fo fackle fhe complicafed issue of gafhering dafa on 
e-resources. A number of issues were idenfified af fhaf 
meeting, many reiferafing fhe poinfs in fhe Jewell reporf. 
Some individuals nofed fhey sfill had liffle dafa. For fhose 
libraries fhaf had dafa, fhe dafa were inconsisfenf and 
unreliable. The libraries fhemselves had a variefy of 
infernal sfrucfures and procedures surrounding fhe 
acquisition and deploymenf of elecfronic resources and 
were nof well organized fo collecf common dafa. Anofher 
major problem idenfified was fhe dearfh of information 
abouf how fhe use of elecfronic information resources 
confribufed fo library user success. 

Meefing parficipanfs agreed fo confribufe funds fo 
address fhis issue and fhe ARE E-Mefrics projecf was 
launched wifh 24 libraries participating. Eed by Rush 
Miller (Universify of Piffsburgh) and Sherrie Schmidf 
(Arizona Sfafe Universify), fhe inifial projecf began in May 
2000 and was complefed in December 2001. Florida Sfafe 
Universify's (FSU) Information Use Managemenf and 
Policy Insfifufe was engaged fo conducf fhe sfudy. If was 
complefed in fhree phases: (1) an inventory of whaf 
libraries were already doing in fhis area and idenfificafion 
of any libraries fhaf could provide besf pracfices; (2) 
idenfificafion and fesfing of dafa elemenfs fhaf could be 
collecfed and used as measures of elecfronic resources for 
frend analysis and benchmarking; and (3) analysis of fhe 
connection befween fhe use of elecfronic resources and 
insfifufional oufcomes. The work of fhis period has been 
described extensively in fhe published liferafure.^ 

A key oufcome of fhe FSU Insfifufe work was a sef of 
recommended sfafisfics and measures (see accompanying 
box). In keeping wifh fhe goals of fhe ARE New Measures 
Inifiafive, documenfs from fhe E-Mefrics projecf are 
available fo fhe wider library communify.^ 

During fhe course of fhis projecf, a Working Group on 
Daf abase Vendor Sfafisfics was esfablished fo investigate 
how fo effectively gafher and reporf dafa from vendor 
dafabases since much of fhe e- resource measure counfs 
would come from fhose sources. ARE builf relationships 
wifh ofher parties inferesfed in vendor sfafisfics as efforfs 
moved forward bofh nationally and infernal tonally. ARE 
continues fo work wifh fhe vendor communify, primarily 
by supporting Projecf COUNTER (Counting Online Usage 
of NeTworked Elecfronic Resources), a multi-agency pro- 
jecf developing a single, internationally accepted Code of 
Practice fhaf will allow fhe usage of online information 
producfs fo be measured more effectively.® Projecf 
COUNTER is now soliciting membership supporf from 
libraries as well as ofher consfifuencies. ARE also moni- 
tors developmenfs wifhin fhe national and international 
sfandards communities as NISO and ISO draff sfandards 
for measuring elecfronic resources are finalized.^ 
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E-Metrics Chronology 

1993 82 libraries report e-resources data for ARL 

Supplementary Statistics 

1996-98 Assessment of ARL Supplemenfary Sfafisfics 
survey dafa on e-resources underfaken by 
Tim Jewell (Universify of Washington) as 
ARL Visifing Program Officer 

1999 ARL New Measures Ref real, e-resources 
identified as one of fhe key areas of concern 

105 libraries reporf e-resources for ARL 
Supplemenfary Sfafisfics 

2000 Two-day meeting of library direcfors and 
senior sfaff focuses on gafhering dafa on 
e-resources 

ARL E-Mefrics Projecf launched wifh 
24 libraries participating; conducfed in fhree 
phases over fwo years by Florida Sfafe 
Universify's (FSU) information Use 
Managemenf and Policy Insfifufe 

Working Group on Vendor Sfafisfics esfablished 

2001 FSU Projecf feam recommends 16 measures of 
e-resources and 3 measures of fhe use or 
performance of fhese resources. 

ARL joins Projecf COUNTER 

2002 Call for participation to ARL libraries to fesf 
fhe measures recommended by FSU feam; 35 
libraries respond and commif to collecf fhe 
dafa for 2001-02 or parf of fhe 2002-03 
academic year and compare resulfs wifh 
vendor dafa 

Gordon Frefwell (Universify of Massachusetts), 
Visifing Program Officer, and Marfha 
Kyrillidou (ARL) collecf and analyze dafa 



E-Metrics Pilot Projects 

Accepfance of fhe E-Mefrics projecf reporf from fhe 
FSU Insfifufe was one step in fhe series of ARL acfivifies 
addressing fhe issue of measuring elecfronic resources. 
Nexf came a call for parficipafion in fhe summer of 2002 
for ARL member libraries fo confinue fhe work of fhe 
projecf and fesf fhe recommended measures in an 
E-Mefrics Pilof Projecf during 2002-03. Thirfy-five 
insfifufions responded, half of fhem new fo fhe projecf 

For fhe 2002-03 E-Mefrics Pilof Projecf, fhe 
parficipafing libraries were asked fo reporf back 
fhrough a sfrucfured quesfionnaire fo Gordon Frefwell 
(Universify of Massachuseffs), who served as an ARL 
Visifing Program Officer. The quesfionnaire gafhered 
information on whaf dafa fhe library was currenfly 
collecting, how much of if could be reporfed for FY 
2001-02, any additional sfafisfics being collected on in- 
house elecfronic resources, and whaf fypes of infernal 
analysis were being done wifh fhe dafa collected. 
Parficipanfs were also asked fo reporf dafa for fhe firsf 
quarfer of fhe fiscal year 2002-03 0uly 2002 fhrough 
September 2003). The goal was fo collecf some in-depfh 
annual dafa from some vendors af some libraries, and a 
wider collection of dafa for a briefer window of acfivify 
(firsf quarfer of currenf fiscal year) from a larger number 
of libraries. Variafions in vendor reporf ing may mean 
fhaf dafa inifially examined would nof be sfricfly 
comparable across vendors. 

An in-person projecf meeting af fhe January 2003 
ALA Midwinter Meefing provided an opporfunify for 
some libraries fo presenf and discuss fhe analysis 
feclmiques and specific oufpuf or producfs fhaf fhey 
have developed and utilized.® As fhe year progressed, a 
number of fhe 2002-03 E-Mefrics parficipanfs spenf fime 
organizing fhe collecfion of "dafabase use" sfafisfics 
(sessions, searches, ifems). The redesigned ARL 
E-Mefrics Web page includes a section where 
parficipafing libraries' spreadsheefs can be seen as 
examples of fhe new E-Mefrics dafa-collecfion process.® 
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Projecf COUNTER issues draff guidelines for 
measuring usage of online information producfs 

Decision reached fhaf dafa being collected 
relafed fo expendifures for e-resources is 
sfable enough fo move from Supplemenfary 
Sfafisfics fo fhe main ARL Sfafisfics as of fhe 
2003-04 collecfion cycle (July 2004) 

Decision reached fo move all ofher E-Mefrics 
dafa elemenfs still being fesfed info fhe 
Supplemenfary Sfafisfics beginning wifh fhe 
2003-04 collecfion cycle (July 2004) 

Call for parficipafion in fhe 2003-04 dafa 
collecfion of E-mefrics issued 



Current Status of Pilot Project and Next Steps 

In lighf of fheir proven ufilify and reliabilify, several dafa 
elemenfs for elecfronic resources previously collecfed 
fhrough fhe ARL Supplemenfary Sfafisfics will be moving 
info fhe regular ARL Sfafisfics dafa-collecfion acfivify 
beginning in July 2004 wifh collecfion of dafa for 2003-04. 
They are ifems relafed fo expendifures on elecfronic 
resources: Compufer Files (one-fime/ monographic 
purchase); Elecfronic Serials; Bibliographic Ufilifies, 
Nefworks, and Consortia (Library); and Bibliographic 
Ufilifies, Nefworks, and Consortia (Exfernal). Ifems 
remaining in fhe ARL Supplemenfary Sfafisfics fhaf are 
continuing fo be fesfed as parf of fhe E-Mefrics Projecf 
include Expendifures on Elecfronic indexes and reference 
fools and Expendifures on Elecfronic full-fexf periodicals. 
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E-Metrics Being Tested 

Patron-Accessible Electronic Resources 

• Number of electronic full-text journals 

• Number of electronic reference sources 

• Number of electronic books 

Use of Networked Resources & Related Infrastructure 

• Number of electronic reference transactions 

• Number of logins (sessions) to electronic databases 

• Number of queries (searches) in electronic 
databases 

• Items requested in electronic databases 

• Virtual visits to library's Web site and catalog 

Expenditures for Networked Resources 
& Related Infrastructure 

• Cost of electronic full-text journals 

• Cost of electronic reference sources 

• Cost of electronic books 

• Library expenditures for bibliographic utilities, 
networks, & consortia 

• External expenditures for bibliographic utilities, 
networks, & consortia 

Eibrary Digitization Activities 

• Size of library digital collection 

• Use of library digital collection 

• Cost of digital collection construction & 
management 

All the data elements collected through the current 
ARL E-Metrics pilot will be moving into the regular ARL 
Supplementary Statistics collection cycle starting in July 
2004 for the year 2003-04. This move will begin to test 
more widely the data collection that thus far has been 
conducted on a pilot-project basis. During 2003-04, the 
pilot process was opened to additional participants. 

Training 

As with all of the new measures projects, training 
was identified by many E-Metrics participant libraries 
as a major need. In addition to providing specific 
information about how to gather the recommended 
statistics, training also should include conceptual 
understanding of the importance of using data for 
decision making and developing a culture of 
assessment that encourages staff to make use of 
gathered data. ARL provides opportunities for 
E-Metrics project participants to share experiences 
at meetings and delivers workshops about gathering 
and using data in a variety of ways.^° 

Conclusion 

Increasingly libraries are called upon to demonstrate the 
educational impact and outcomes achieved because of 



the increased availability of electronic resources. In the 
new measures environment, we are faced with the 
challenge of identifying (a) new "species" of metrics and 
(b) new methods for measuring the effectiveness of these 
new species. The work described above primarily focuses 
on the identification of new species, i.e., new data 
categories that will give us some general sense of the 
evolution of electronic resources in terms of input and 
output measures. There are also efforts imderway, such as 
the MINES Project described by Franklin and Plum in this 
issue and the effort to develop e-QUAL for the digital 
library environment, that are attempting to develop new 
user-based methods for understanding the value derived 
from the use of electronic resources. ARL continues to 
seek and develop the best measures to demonstrate how 
the provision of electronic resources contributes to library 
users' success. 

1 ARL Supplementary Statistics 2001-02 (Washington, D.C.: ARL, 2003). 

2 A printed compilation of the investigator reports and the instruction 
manuals are available for purchase and all the documents are on the 
E-Metrics Web site <http: / / www.arl.org / stats / sup / >. 

^ A full report was sent to each ARL member library as part of fhe report 
on the 1995-96 ARL Supplementary Statistics and is also available at 
<http: / / www.arl.org/ stats/ specproj / etrends.htm>. 

^ Rush Miller and Sherrie Schmidt, "E-Metrics: Measures for Electronic 
Resources," in Proceedings of the 4th Northumbria International Conference 
on Performance Measurement in Libraries and Information Services 
(Washington, D.C.: ARL, 2002): 37-42; Wonsik "Jeff" Shim, Charles R. 
McClure, and John Carlo Bertof, "Preliminary Sfatistics and Measures 
for ARL Libraries to Describe Electronic Resources and Services," in 
Proceedings of the 4th Northumbria International Conference on Performance 
Measurement in Libraries and Information Services (Washington, D.C.: 
ARL, 2002): 337-344; Wonsik "Jeff" Shim, Charles R. McClure, and 
John Carlo Bertot, "Measures and Sfatistics for Research Library 
Networked Services: ARL E-Metrics Phase II Report," ARL Bimonthly 
Report 219 (Dec. 2001): 8-9,<http:/ / www.arl.org/newsltr/ 

219 / emetrics.html>; Wonsik "Jeff" Shim, Charles R. McClure, and 
John Carlo Bertot, "Data Gathering Practices in the Networked 
Environment," ARL Bimonthly Report 213 (Dec. 2000): 6-8 
<http:/ /www.arl.org/newsltr/213/data.html>; Wonsik "Jeff" Shim 
and Charles R. McClure, "Measuring Services, Resources, Users, and 
Use in the Networked Environment," ARL Bimonthly Report 210 (June 
2000): 9-10,<http:/ /www.arl.org/newsltr/210/emetrics.html>; Julia 
C. Blixrud, "Measures for Electronic Use: The ARL E-Metrics Project," 
in Statistics in Practice— Measuring and Managing: Proceedings ofIFLA 
Satellite Conference, Loughborough, England, August 2003 
(Loughborough: Library & Information Statistics Unit (LISU), 
Loughborough University, 2003): 73-84, <http: / / www.lboro.ac.uk / 
departments/ dis/lisu/statsinprac.html>. 

^ <http: / / www.arl.org / stats / newmeas / emetrics / >. 

® <http:/ / www.projectcounter.org/>. 

^ NISO Z39.7-2002 Draft Standard for Trial Use: Information Services 
and Use: Metrics & Statistics for Libraries and Information 
Providers — Data Dictionary, <http: / / www.niso.org / emetrics / >. 

® The meeting was hosted by the University of Pennsylvania, which 
has been developing the Penn Library Data Farm, a repository of 
quantitative information to aid in the measurement and assessment 
of library resource use <http: / / metrics.library.upenn.edu / 
prototype / about / >. 

® Like every Web-redesign it is work in progress located at 
<http: / / www.arl.org / stats / newmeas / emetrics / >. 

ARL is offering a Web casf in October 2003 on data-collection 
changes, <http:/ /www.arl.org/arl/workshops.html>. 
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Research Library Involvement 
IN Learning Outcomes 
Assessment Programs 

by Sheila Young, Science Reference Librarian! Engineering, 
Arizona State University, and Julia C. Blixrud, Assistant 
Executive Director, External Relations, ARE 

H ow could research libraries participate in the 
assessment of learning outcomes at the 
institutional level and determine how to measure 
their contribution to the learning process? This is the 
question now being pursued by the ARL Learning 
Outcomes Working Group. In 2000, Eller Distinguished 
Service Professor Kennefh Smifh, Universify of Arizona, 
prepared a while paper for ARL fhaf discussed how a 
library mighf begin developing shared learning oufcomes 
wifh fhe larger campus communify and fhen creafe 
"offerings" based on fhose shared oufcomes, fo infegrafe 
info fhe curriculum.^ Dr. Smifh recommended fhaf 
libraries "undersfand fhe processes fhaf are used fo 
define learning oufcomes, fo selecf measures, fo 
collaborafe wifh ofher academic deparfmenfs, and fo use 
fhe resulfs fo improve fheir programs." 

The Learning Oufcomes Working Group, initially 
chaired by Carla Sfoffle, Dean of Libraries af fhe 
Universify of Arizona, was esfablished fo follow up on 
Dr. Smifh's recommendafions. The working group is 
currenfly led by Karen Williams, Team Leader for 
Undergraduafe Services, Universify of Arizona Library. 
As wifh ofher New Measures Inifiafive projecfs, fhe 
working group was charged fo idenfify exisfing measures 
fhaf could be builf upon and applied in libraries. A scan 
of fhe higher educafion environmenf found fhaf, while 
fhe emphasis on esfablishing learning oufcomes and 
assessing fhose oufcomes has gained momenfum in 
recenf years, campuses are sfill in various sfages of 
grappling wifh fhis issue. Given fhis, fhe working group 
decided fo develop fheir own while paper fhaf would 
describe fhis environmenf and, using if as a foundafion 
for undersfanding fhe currenf sfafe of research and 
implemenfafion in fhis field, decide how fo fake Dr. 
Smifh's recommendafions a sfep furfher. 

The working group's while paper, now in draff, 
examines fhe definifions and sources of learning 
oufcomes, domains of learning, levels of assessmenf, 
and frameworks suggesfed for puffing fhese pieces 
fogefher wifhin an insfifufion. Whaf follows is a 
summary of fhe working group while paper. 




Definitions of "Learning Outcomes" 

Numerous authors express the concept of sfudenf 
learning oufcomes as fhe changes fhaf occur wifhin fhe 
sfudenf, nof whaf fhe insfrucfor does.^ Thai is, learning 
oufcomes are "a level of knowledge, skills, abilities fhaf 
a sfudenf has affained."^ This emphasis on fhe change 
fhaf fakes place wifhin fhe sfudenf is also reflecfed in a 
definifion developed by Baffersby and embraced by 
members of fhe working group. 

"Learning oufcomes are fhe essenfial and 
enduring knowledge, abilities (skills), and 
affifudes (values, dispositions) fhaf consfifufe 
fhe infegrafed learning needed by a graduafe of 
a course or program."^ 

Sources of Learning Outcomes: National 

Regional accrediting agencies are one source of the 
criteria upon which an institution will be evaluated. 
Accreditation criteria have been recently revised 
reflecting an increased emphasis on assessment and 
institutions are beginning to gain experience with 
these new expectations. Lindauer has examined and 
analyzed documents of the accreditation agencies and 
evaluates trends and specific text related to libraries. 

Her work presents examples from these documents 
and makes five observations and recommendations for 
library action. These concern the role of the library in 
information literacy, the role of the library as an 
"academic support unit," librarian involvement in 
accreditation standards, promotion on campus by 
librarians of the ACRL information literacy documents, 
and the need for librarians to stay informed about the 
current assessment projects of the ACRL, ARL, the 
Association of American Higher Education, and other 
organizations involved with assessment.^ 

Discipline-specific documents are an additional 
source of learning outcomes that are valued by 
departments across campuses. On a national level, 
the American Chemical Society, the American 
Psychological Association, and the Accreditation Board 
for Engineering and Technology Inc. have all developed 
such statements. The Council for Higher Education 
Accreditation (CHE A) lists a directory of "Specialized 
and Professional Accrediting Organizations" that are 
either recognized or will undergo a review by them.® 
The accreditation documents of these professional 
organizations periodically undergo revision and this 
may provide librarians an opportunity to contribute 
to the new standards. 

Another source of learning outcomes from a 
national body is a 1995 report from the U.S. Department 
of Education, Office of Educational Research and 
Improvement: "National Assessment of College Student 
Learning: Identifying College Graduates' Essential Skills 
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in Writing, Speech and Listening, and Critical Thinking: 
Final Project Report."^ The report identifies "higher 
order communication and thinking skills that college 
graduates should achieve" and describes the skills of 
analyzing, locafing, and evaluating information as 
dimensions of crifical fhinking. 

Wifhin fhe library communify, ACRL has produced 
information liferacy and compefency sfandards for 
higher educafion® and a sfandards fool kif^ fhaf lisfs 
fhe oufcomes for each sfandard. In addifion, an ACRL 
"Besf Practices" inifiafive has 
developed "Characferisfics of 
Programs of Informafion 
Liferacy fhaf lllusfrafe Besf 
Practices: A Guideline," which 
was approved by fhe ACRL 
Board in 2003.^° All of fhis work 
is useful as librarians examine 
fhe needs and challenges fhaf 
are particular fo fheir own insfifufions. 

Sources of Learning Outcomes: Local 

Sfafemenfs by insfifufional governing boards indicafe 
valued oufcomes, including learning oufcomes, if 
presenf. Examining fhe documenfs produced by fhe 
governing board, insfifufion (e.g., fhe mission and fhe 
sfrafegic plan), and fhe individual colleges, 
deparfmenfs, and programs will indicafe fhose learning 
oufcomes fhaf are of greafesf imporfance. 

Integration of Learning Outcomes 
through All Levels 

Writing learning outcomes that are internally consistent 
with the department or discipline, the institution, state, 
and accrediting agency requires an examination of 
documents from all these levels. Huba describes this 
process of integration as a triangular process. She 
describes the relationship of a "design backward" 

(from the broadest learning outcome of the institution 
or discipline to the most specific learning outcomes of 
an individual lesson) and the "deliver forward" 
experience (from the students' perspective). Since 
many institutions are in various stages of developing 
learning outcomes at each level, librarians have many 
opportunities to participate in the process of integrating 
learning outcomes through all levels of the institution. 

Assessment 

Assessing learning outcomes is an exceptionally 
complex process and, to guide the process, certain 
fundamental questions need to be answered: the 
purpose, the content, and the level of assessment all 
give direction to the development of assessment 
measures. A number of models for assessing learning 
outcomes are being developed and tested on different 
campuses. Recent literature on higher education 



assessment illustrates that some common themes and 
learning objectives are arising among the general 
educational outcomes being tested and used. A key 
resource for monitoring developments in this rapidly 
evolving area is Assessment Update, a journal edited by 
Trudy W. Banta, Vice Chancellor for Plarming and 
Institutional Improvement at Indiana University- 
Purdue University at Indianapolis.^^ 

In order to assess how library programs and 
services contribute to student learning outcomes at the 

student, department, or 
institutional level, librarians 
need to inform themselves on 
how learning outcomes 
assessment is practiced on their 
own campus. In addition, the 
contributions of the library to 
learning outcomes could be 
described in a broader context 
of the students' experiences with the library; not just 
those limited to the instruction activity. 

Since the library has multiple roles such as service, 
support for research, and instruction, these roles may 
need to be addressed separately and by different means 
of assessment. For example, the LibQUAL+^''^ project 
addresses assessment of the quality of service provided 
by a library while the Assessing ILL/DD project, as well 
as the MINES methodology proposed to document 
usage patterns of networked electronic services, 
both contribute to an assessment of library support 
for research. 

In the movement toward learner-centered 
education, the emphasis on the student and on learning 
outcomes is also growing in importance as part of the 
evaluation of academic programs. Evidence of the 
contribution of the library to student learning outcomes 
is therefore an important aspect of demonstrating the 
value of the library to academic programs and the 
institution. 

Measures 

Developing measures to document how the library 
contributes to the learning outcomes at the department 
or institutional level has been challenging. Focusing on 
the course level, a project by ACRL resulted in a report 
on how librarians in 23 institutions are working with 
faculty in assessing information literacy learning 
outcomes. In a study to evaluate an information 
literacy program, librarians who worked with faculty in 
the English department noted that the collaboration 
with the English faculty increased awareness that 
"information literacy standards are remarkably similar 
to the standards of critical thinking they try to impart in 
freshman English."^^ This recognition of the 
relationship of critical thinking and information literacy 



Recent literature on higher education 
assessment illustrates that some common 
themes and learning objectives are 
arising among the general educational 
outcomes being tested and used. 
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provides a foundation upon which to build mutual 
learning outcomes. 

Measures can be categorized as direct or indirect 
and within each of these groups, the measures can be 
quantitative or qualitative. Indirect measures may not 
be adequate alone, but may enrich or illuminate aspects 
of what direct measures say 
about levels of academic 
achievement. 

One indirect measure that 
has been used by institutions is 
the College Student Experiences 
Questionnaire (CSEQ). This 
survey asks students to indicate 
the amount of time they spend 
engaged in various activities 
that affect learning. It includes 
eight questions regarding 
library use. Analysis of the responses shows that 
"focused library activities" have a "significant" impact 
on self-reported development of critical thinking.^^ 
Others found that "students who perceive that their 
campus emphasizes information literacy gain more in 
this area, net of other influences."^® The same 
researchers also found that the "library experiences of 
undergraduates are related to select educational 
purposeful activities" and that the library "appears to 
be a positive learning environment." However, the 
results also indicated "the library use does not appear to 
directly contribute to gains in information literacy and 
other desirable outcomes." The authors discuss the 
following possible explanations: CSEQ may not be the 
best measure for information literacy, "lack of baseline 
measures" and "a variety of experiences during 
college... contribute to gains and satisfaction, not just 
one type of experience." 

At the system level, the California State University 
has launched a three-phase assessment of information 
literacy skills of the students at all 23 campuses. This 
project uses both qualitative and quantitative measures 
and employs multiple methods of assessment. Qne part 
of the inquiry includes questions related to information 
scenarios as well as sets of questions scaled to areas 
related to information competence "such as library use, 
research process skills, achievement, presence of 
reference materials in the home, and computer and 
media literacy." The "information scenarios were 
constructed to elicit evidence of information 
competence when applied to general knowledge, rather 
than discipline-based knowledge." Responses are 
measured in terms of "Breadth" (number of different 
ideas) and "Depth" (number of discrete ideas). In a 
preliminary report, the author finds that the students 
that self-reported difficulty finding and using 
information tended to have lower Breadth and Depth 



scores. The study also found that "the greater the usage 
of the library resources, the greater the Breadth and Depth 
of the student's responses to the scenarios. 

The SAILS Project, at Kent State University and ARL, 
seeks to develop a standardized test "to measure 
information literacy skills, gather national data, provide 

norms, and compare information 
literacy measures with other 
indicators of student 
achievement." The instrument, 
based on the ACRL standards, is 
"for programmatic level 
assessment of information literacy 
skills that is valid and thus 
credible to university 
administrators and other 
academic personnel."^® It is 
possible that the SAILS instrument 
could be used as a direct measure of information skills 
development. It would then be necessary to link this to the 
learning outcomes at the department, program, or 
institution level.^® 

Performance Indicators 

Lindauer identifies several performance indicators that 
might be used as measures. For example, 

• "Perceptions of recent graduates about their 
information literacy skills training /experience from 
undergraduate study contributes to their success in 
graduate/ professional programs." 

• "Success in applying information literacy skills on 
the job as perceived by alumni and employers." 

The same author suggests that "libraries could 
strengthen their evidence by triangulating data and 
supporting information. They could group such measures 
as selective qualitative descriptions of resources and 
services with usage data for specific services and with 
user satisfaction and perceptions of benefit from survey 
findings. All this could be grouped to show the impact of 
library and information resources and services..."^® 

Conclusion and Next Steps 

The Learning Qutcomes Working Group white paper 
brings together more detail about these and other 
resources on the state of assessment of learning outcomes 
especially as they relate to libraries. The goal is that this 
environmental scan of available research and experience 
will enable the working group to focus on identifying the 
most promising mechanisms or instruments that would be 
useful in demonstrating the research library's direct or 
indirect impact on learning outcomes at the department or 
institutional level. A draft of the white paper will be 
available on the ARL Web site by the end of the calendar 
year. For more information, contact Sheila Young 
<sheila.young@asu.edu> or Julia Blixrud <jblix@arl.org>. 



In order to assess how library programs 
and services contribute to student learning 
outcomes at the student, department, or 
institutional level, librarians need to 
inform themselves on how learning 
outcomes assessment is practiced 
on their own campus. 
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Evaluating the Educational 
Impact of Digital Libraries 

hy Martha Kyrillidou, Senior Program Officer for Statistics 
and Measurement, ARL 

T he National Science Digital Library (NSDL) 
Educational Impact and Evaluation (El&E) 
Standing Committee is developing a strategy for 
evaluating the educational impact of digifal libraries. 

As in all new measures work, fhe lack of common 
definifions or understood vocabulary is a challenge. 
Concepfually, however, assessing educafional impacf is 
frying to answer fhe quesfion, "Whaf difference does if 
make?" in ferms of oufcomes achieved (e.g., changes in 
behavior, affifudes, skills, knowledge). Quesfions of 
impacf are difficulf to measure and fhey usually require 
exfensive and long-term assessmenf efforfs. 

What is NSDL? 

NSDL is a National Science Foundation (NSF) program 
with over 100 projects.^ It is a network of digifal libraries 
and services fhaf are fargefing differenf audiences. A 
large componenf of fhis nefwork is fhe core integration 
efforf fhaf affempfs fo creafe shared fechnologies, proto- 
cols, a cenfral reposifory, and a core porfal (NSDL.org). 
The mission of NSDL is (a) fo encourage and susfain con- 
tinual improvemenfs in fhe qualify of science, fechnology, 
engineering and mafhemafics (STEM) education for all 
sfudenfs, and serve as a resource for lifelong learning and 
(b) a collective efforf fo build a national digifal library of 
high qualify STEM educafional maferials for sfudenfs and 
teachers af all levels, in bofh formal and informal settings. 
ARL and Texas A&M are parfners in one of fhe NSDL 
projecfs funded by NSF fo explore fhe developmenf of 
an evaluation fool for NSDL. (See fhe arficle on fhe ARL 
e-QUAL projecf on page 8 of fhis issue.) 

Evaluation in the Context of NSDL 

Evaluation of NSDL has been primarily formative in nafure 
wifh differenf projecfs and parficipanfs bringing forward 
fheir own assessmenf activities. In addition, fhere is contin- 
uous NSF program evaluation and encouragemenf fo 
broaden participation of ofher NSF programs fhaf supporf 
fhe developmenf of assessmenf capacify. 

The emphasis of fhe EI&E Committee is fo develop a 
communify for evaluafing digifal libraries af a variefy of 
levels and using diverse models. To advance fhis process a 
workshop on "Developing an Evaluation Sfrafegy for fhe 
Educational Impacf of fhe National Science Digifal Library" 
was held in Washington, D.C., on October 2-3, 2003. ARL's 
e-QUAL projecf was represented and a reporf on EI&E 
progress will be developed for fhe ARL communify. 



V' It it- 



1 Lee Zia, "The NSF National Science, Technology, Engineering, and 
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Project SAILS: Standardized 
Assessment oe Ineormation 
Literacy Skills 

by Jidia C. Blixrud, Assistant Executive Director, External 
Relations, ARE 

I nformation literacy is the ability to "recognize 
when information is needed and... locate, evaluate, 
and use effectively the needed information."^ This 
skill set is developed through information literacy 
programs operated by libraries primarily, but not 
exclusively, for undergraduates 
in academic institutions. Are 
these programs successful? Do 
they advance the skills of 
students? How can we measure 
these programs' success? Kent 
State University and ARL have 
partnered on Project SAILS 
(Standardized Assessment of 
Information Literacy Skills), 
which is developing an 
instrument for programmatic 
level assessment of information literacy skills that is 
valid and thus credible to university administrators 
and other academic personnel. 

Several years ago, librarians at Kent State 
University Libraries & Media Services realized that, 
if university administrators were to be persuaded to 
allocate resources necessary to grow the information 
literacy program, librarians must be prepared to 
substantiate the claim that these skills indeed make a 
difference institutionally. They must be able to answer 
such questions as: What are students' entry skills 
upon admission to the university, and is there a 
significant change in skill levels from the freshman 
year to graduation? If there are significant changes 
in students' information literacy skills, do those skills 
then have any correlation to student's academic 
success and retention? The librarians conducted a 
thorough search of the library literature and 
determined that the profession was not yet in a 
position to agree upon the best method for assessing 
information literacy skills, let alone assert that those 
skills make a difference. 

Librarians need a tool to measure information 
literacy that could be standardized, is proven to be 
valid and reliable, contain items not specific to a 
particular institution or library but assesses at the 
institutional level, can be administered easily, and 
provides for both external and internal benchmarking. 
With such a tool, a library could measure information 
literacy skills, gather national data, provide norms, 
and compare information literacy measures with other 
indicators of student achievement. Libraries would be 



able to document information literacy skill levels, 
establish internal and peer benchmarks of 
performance, pinpoint areas for improvement, 
identify and justify resource needs, and assess and 
demonstrate effect of changes in their instructional 
programs. 

Using the Wisconsin Ohio Reference Evaluation 
Project (WOREP) as inspiration, Lisa O'Connor, 
Instructional Services Coordinator; Carolyn Radcliff, 
Head of Reference Services; and Julie Gedeon, 
Manager of Academic Technology Services Evaluation 

at Kent State University 
Libraries & Media Services 
began using the process of 
systematic instructional 
design to develop such a tool 
in 1998. They chose to use 
item response theory (IRT) 
as the measurement model 
on which to create a new 
instrument.^ 

A set of questions of 
varying difficulty level is 
being produced to measure information literacy. 

Each question addresses one of the model learning 
outcomes as identified by the ACRE Information 
Literacy Competency Standards for Higher Education.^ 

Use of these outcomes will maximize the applicability 
of the instrument to a wide variety of academic 
institutions for internal and external benchmarking. 
The questions are tested with individual students, in 
small groups, and in field trials (which closely emulate 
actual survey administrations). 

While the SAILS survey is intended to be 
delivered in electronic form over the Web, it can be 
delivered using paper and pencil. Using the Web, a 
random set of about 45 questions drawn from the 
instrument's data bank of now approximately 150 
items are delivered at the time the test is administered 
to a student. Project participants can choose the most 
appropriate testing population for their institution 
as long as the students are undergraduates. 

Institutions can test students in one of three 
situations: 

• Highly monitored: The test-taking is proctored 
and the procedures are consistent across all 
students. Students check in and out of a 
specified testing site and the proctor is aware of 
the testing. A variation of this is when students 
take the test as a group during class time. 

• Loosely monitored: Students come to a specified 
testing site, but do not check in or out. 

• Unmonitored: Students take the test anytime 
and from anywhere. 



... if university administrators were to 
be persuaded to allocate resources 
necessary to grow the information 
literacy program, librarians must be 
prepared to substantiate the claim that 
these skills indeed make a difference 
institutionally. 
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The answers are collected and analyzed using 
WINSTEPS, a modeling program created by researchers 
at the Mesa Institute at the University of Chicago. 

The Kent State University staff received a granf in 
fall 2002 from fhe Insfifufe of Museum and Library 
Services (IMLS) for Projecf SAILS. The fhree-year granf 
is supporfing confinued developmenf of fhe insfrumenf 
and fesfing af ofher insfifufions. If will also enable 
projecf sfaff fo creafe fhe foundafion of a resulfs 
dafabase fhaf will address infernal and exfernal 
benchmarking. The granf also includes funding for fhe 
commissioning of fhree discipline-specific assessmenf 
modules, which will serve as a model for fufure 
developmenf of ofher discipline-specific modules. 

Since fhere is significanf overlap between student 
learning outcomes and information literacy, ARL's 
Learning Outcomes Working Group recommended that 
the SAILS Project become one of fhe projecfs under fhe 
ARL New Measures Inifiafive. The ARL Sfafisfics and 
Measuremenf Commiffee endorsed SAILS and 
posifioned if as an acfivify wifhin fhe Learning 
Oufcomes Working Group agenda. As wifh ofher 
New Measures projecfs, parficipafing insfifufions pay 
fees fo cover some of fhe cosfs of fool developmenf 
and dafa reporfing. 

Projecf SAILS is a collaborafion befween fhe 
Information Services and fhe Libraries & Media Services 
divisions af Kenf Sfafe Universify. Mary Thompson, 
Inferim Liaison Librarian for Business, serves as Projecf 
Coordinator; Julie Gedeon manages dafa measuremenf 
and sfafisfics; Lisa O'Connor is in charge of fesf 
developmenf; Carolyn Radcliff handles fesf 
adminisfrafion; and Rick Wiggins is fhe Web 
Programmer. In addition fo projecf parficipanfs, an 
advisory council provides inpuf for fhe survey ifems. 

ARL is responsible for coordinafing and managing 
parficipafion for all insfifufions whefher fhey are ARL 
members or nof. This includes calls for parficipafion; 
making arrangemenfs for planning meetings, framing 
workshops, and follow-up meetings; and marketing and 
public relafions for fhe projecf. Julia Blixrud is fhe ARL 
liaison for Projecf SAILS. 

The IMLS-funded activities are divided info fhree 
phases. Each phase is examining bofh fhe survey ifems 
as well as fesf adminisfrafion. Parficipanfs are also 
closely involved in working wifh Projecf SAILS sfaff fo 
develop dafa reporfs. Phase I was conducfed in spring 
2003 af seven insfifufions: Universify of Arizona, 
Auburn Universify, Oregon Sfafe Universify, San Jose 
Sfafe Universify, Virginia Polytechnic Insfifufe and Sfafe 
Universify, Washington Sfafe Universify, and Weber 
Sfafe Universify. Represenfafives of fhose insfifufions 
mef in June 2003 fo review fhe resulfs of dafa analysis 
and fo share experiences regarding fesfing sifuafions. 



Phase II covers fhe fall 2003 and spring 2004 
semesters and over 40 insfifufions have agreed fo 
parficipafe. 

Phase III is scheduled for fall 2004 and spring 2005, 
wifh a projecfed parficipafion of 100 libraries. A call for 
parficipafion for Phase III will be issued in January 2004. 

The rapidify by which insfifufions signed up for 
Projecf SAILS and fhe enfhusiasm for parficipafion 
signals fhaf libraries are looking for fools fhaf can help 
fhem defermine fhe effecfiveness of fheir informafion 
lifer acy programs. If is anofher example of how fhe 
leadership of one library working collaborafively wifh 
ofher inferesfed libraries can creafe a useful, 
sfandardized fool fo benefif fhe larger library 
communify. 

More information on Project SAILS can he found at 
<http:llwww.proiectsails.orgl>. 

1 American Library Association, Presidential Committee on 
Information Literacy, "Final Report" (Chicago: American Library 
Association, 1989), <http: / / www.ala.org / Content / 
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/Presidential_Committee_on_Information_Literacy.htm>. 

2 Lisa G. O'Connor, Carolyn]. Radcliff, and Julie A. Gedeon, 
"Assessing Informafion Literacy Skills: Developing a Standardized 
Instrument for Institutional and Longitudinal Measurement," in 
Crossing the Divide: Proceedings of the Tenth National Conference of the 
Association of College and Research Libraries (Chicago: ACRL, 2001): 
163-174, <http:/ / www.ala.org/Content/NavigationMenu/ 

ACRL /Events_and_Conferences/oconnor. pdf >; and Lisa G. 
O'Connor, Carolyn J. Radcliff, and Julie A. Gedeon, "Applying 
Systems Design and Item Response Theory to the Problem of 
Measuring Information Literacy Skills," College b Research Libraries 
62, no. 6 (Nov. 2002): 528-543. 
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Documenting Usage Patterns oe 
Networked Electronic Services 



First Page of Survey 



Networked Electronic Services 
Library User Survey 



This survey is being conducted by the University in order 
to assess usage of the Library's electronic services. 




by Brinley Franklin, Director of Libraries, University of 
Connecticut, and Terry Plum, Assistant Professor, Simmons 
Graduate School of Library and Information Sciences 

A new initiative now being considered by the ARL 
Statistics & Measurement Committee is a Web- 
based survey that a library could administer to 
collect reliable data on the usage of networked 
electronic services from both within the library and 
remotely through the Web. The idea is to build on the 
experience of sfudies conducfed af five academic healfh 
science libraries and fwo main libraries befween 1999 
and 2003. In fhose sfudies, more fhan 15,000 library 
users were surveyed as fhey accessed fheir libraries' 
nefworked elecfronic services. The survey documenfed 
why pafrons were using elecfronic resources in fhose 
libraries and fracked fhe differences befween in-house 
and Web usage, comparing fhe location of fhe user and 
fheir status (graduafe sfudenf, faculfy, undergraduafe, 
efc.) wifh fhe purpose of the use. 

The mefhodology measured bofh in-house and 
remofe usage in a way fhaf allowed fhe cosfs of 
providing nefworked elecfronic services fo be 
disfribufed among fhe universify's primary functions. 

In fhis sfudy, fhose functions were defined in four 
cafegories: sponsored (funded) research, insfrucfion/ 
education /deparfmenfal research, patienf care, and 
all ofher activities. The survey is a pop-up screen 
adminisfered randomly fo users of nefworked elecfronic 
services fhroughouf fhe year using a sfafistically valid 
sampling plan. 

The sfudy, now called "Measuring fhe Impacf of 
Nefworked Elecfronic Services" (or MINES), was 
designed and underfaken by fhe aufhors. The 
researchers concluded fhaf alfhough fhe compufing 
environmenfs and survey implemenfafions in each 
library were differenf, fhe mefhodology and overall 
approach was workable af each library. As a resulf, 
fhey are confidenf fhaf fhe dafa collecfed are somewhaf 
comparable, recognizing fhaf fhere will always be 
distinguishing characferisfics driven by fhe local user 
population and insfifufional characferisfics. 

Experience also led fhe researchers fo conclude 
fhaf fhe mosf effecfive means fo meef fhe Web survey 
criferia was fo run all access for nefworked elecfronic 
resources fhrough a gafeway fhaf aufhenficafed access 



All responses are confidential. 

After completing the survey, you will be connected 
to the service you selected. 

Thank you for your help. 



Patron Status: 

Affiliation: 

Location: 




Purpose 
for Using 
Online 
Resources: 



(O) A. Sponsored (Funded) 
Research 
—Definition 

@ B. Instruction/Education/ 
Departmental Research 
—Definition 

(5) C. Patient Care 
—Definition 

@ D. All Other Activities 
—Definition 



Please fill out Status, Affiliation, 
Location, and Purpose Fields. 



Submit 

Response 



and passed on the request. Such an arrangement 
prevents lost data due to bookmarks, non-library Web 
pages, and other non-library routes to access library 
networked electronic services. 

Initial use of fhe mefhodology produced a number 
of findings useful for guiding decisions on how fo 
manage nefworked elecfronic resources. For example, 
resulfs from fhe firsf four academic healfh science 
libraries showed fhaf: 

• There were approximafely four remofe 

nefworked elecfronic users for every in-house 
user. 
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• Remote users were demographically different 
from in-library users; fhe 4-1 rafio was even 
higher for fhe cafegory of users who idenfified 
fhemselves as faculfy/sfaff/ research fellows. 

• The purpose of use reporfed for remofe usage of 
nefworked elecfronic resources was significanfly 
differenf from in-house usage. For example, 
fhose using nefworked elecfronic resources fo 
conducf sponsored research do so mosf 
infensively from on-campus buf nof from in fhe 
library. Those using fhese services for 
insfrucf ion / educaf ion / non-sponsored research 
were more likely fo be in fhe library fhan people 
using fhe resources for ofher purposes. 

• Pafrons using elecfronic services for sponsored 
research represenfed 34% of fhe usage in fhe 
healfh science libraries, as compared fo 16% 

in fhe fwo main libraries. 

The sfudy also found fhaf, despife proxy servers, 
modem pools, and similar remofe enabling services, 
fhe pafrons being surveyed wenf fo fhe universify fo use 
fheir library's nefworked services. Thaf is, when fhey 
used fhese services fhey were eifher on campus or in 
fhe library many more fimes fhan off-campus. 

However, fhe users' purposes for using elecfronic 
resources ranged significanfly among fhe four libraries, 
underscoring fhe need fo consider local conditions 
as libraries have differenf characferisfics and user 
populations. 

The aufhors concluded fhaf MlNES's Web-based 
survey mefhodology employed af fhe seven libraries 
could serve as a model for similar user sfudies af ofher 
libraries. The discussions now underway wifhin fhe 
ART Sfafisfics and Measuremenf Commiffee focus on 
how fo fake a nexf sfep fo apply fhis mefhodology fo 
fhe nefworked elecfronic services offered by a more 
complefe complemenf of research libraries. 

A full description of fhe research projecf and ifs 
findings was firsf delivered af fhe Norfhumbria Life 
conference, parf of IFLA in Glasgow in 2002. See 
Brinley Franklin and Terry Plum, "Nefworked 
Elecfronic Services Usage Patterns af Four Academic 
Healfh Sciences Libraries," Performance Measurement and 
Metrics 3, no. 3 (2002): 123-133, <hffp:/ /www. arl.org/ 
sfafs /newmeas / emefrics / Franklin_081102.pdf>. 
Subsequenf findings were recently presented by the 
authors in September 2003 at the Conference on Users 
in fhe Elecfronic Information Environmenf in Espoo, 
Finland. 

— Copyrighf 2003 Brinley Franklin and Terry Plum 



Cost Effectiveness of 
Library Operations 
AND Services 



Assessing ILL/DD Services Study: 
Initial Observations 

by Mary E. Jackson, Director of Collections and Access 
Programs, ARE 

W haf are fhe characferisfics of a cosf-effecfive 

inferlibrary loan (ILL) and documenf delivery 
(DD) operation? To gef a clearer answer fo 
fhis question, ARL has underfaken a fhird sfudy of fhe 
performance of ILL/DD operations in Norfh American 
libraries. The currenf sfudy. Assessing ILL/DD 
Services, collecfed dafa on fhe 2001-02 performance of 
mediafed and user-inifiafed ILL/DD operations in 
72 research, academic, and special libraries. 

New fo fhis sfudy is a focus on user-inifiafed, 
or unmediafed, ILL services. User-inifiafed ILL is 
defined, for fhis sfudy, as requesfs fhaf are initiafed 
wifhouf ILL sfaff involvemenf and received direcfly by 
a pofenfial supplier. This is fhe firsf comprehensive 
sfudy of a range of user-inifiafed services: INNReach, 
URSA, Loansome Doc, ILLINET Online, RAPID, and 
user-inifiafed use of commercial documenf delivery 
suppliers. In addition, fhe projecf collecfed dafa on fhe 
performance of local documenf delivery services in 
25 libraries. 

The projecf measured unif cosf, fill rafe, and 
furnaround time for mediafed and for user-inifiafed 
inferlibrary borrowing and lending. Cosf dafa includes 
fhe cosfs for sfaff, nefwork/ communications, delivery, 
photocopying, supplies, equipment, and borrowing fees. 
Turnaround fime for borrowing, as in fhe previous 
sfudy, was measured in calendar days from fhe date fhe 
user submiffed fhe requesf fo fhe dafe ILL sfaff notified 
fhe user fhaf fhe item was available for pickup (or 
mailed fhe item fo fhe user). Turnaround fime for 
lending, a measure nof included in fhe previous sfudy, 
capfures fhe elapsed fime between a library receiving 
and shipping a request. 

When the cost data are combined with performance 
dafa on furnaround fime and fill rafe, if is possible fo 
idenfify fhe characferisfics of fhe mosf cosf effective 
inferlibrary borrowing and lending operations and fo 
showcase good practices fhaf could benefif ofher 
libraries. 

Results from Previous ILL/DD Studies 

In 1992, ARL and fhe Research Libraries Group (RLG) 
collaborafed in a joinf projecf fo collecf defailed 
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Assessing ILL/DD Services Study 

Results for ARE Library Participants, Mean Values 

Borrowing Lending 





2002 


1996 




2002 


1996 


Unit cost. Mediated 


$17.50 


$18.35 




$9.28 


$9.48 


Unit cost. 
User-initiated 


$2.04-$14.70 


NA 




$3.47-$12.06 


NA 


Fill rate. Mediated 


86% 


85% 




58% 


58% 


Fill rate. 
User-initiated 


84-91% 


NA 




76%-88% 


NA 


Turnaround time. 
Mediated 


9.29 days (loans) 

& 6.23 days (copies) 


16.9 days (loans) 

& 14.9 days (copies) 




1.75 days (loans) 

& 1.37 days (copies) 


NA 


Turnaround time. 

User-initiated 

(loans) 


2.30-8.23 days 


NA 




0.57-1.45 days 


NA 



information on 1991 costs incurred by research libraries 
for ILL transactions^ That study found that research 
libraries spent an average borrowing unit cost of $18.62 
and lending unit cost of $10.93. 

In 1995, ARL conducted a second study that built on 
the 1992 study by measuring not only the cost but also 
several performance attributes of ILL/DD operations in 
research libraries and, for comparison purposes, in 
college libraries.^ This study found that research libraries 
spent an average of $18.35 on a borrowing transaction, 
$9.48 on a lending transaction, and took an average 15.6 
calendar days to receive a borrowing request. 

The Current Results 

The accompanying chart presents the current findings 
for the 59 participating ARL libraries and compares the 
results with those for the 97 ARL participants in the 
1996 study. It is important to remember that different 
libraries participated in both studies, which may result 
in discrepancies when comparing the two different 
studies. Additional analysis will be undertaken to 
compare the results of the 46 libraries that participated 
in both the 1996 and 2002 studies and the 29 libraries 
that participated in all three ARL ILL/DD studies. 

The current study confirms informal, institution- 
specific studies and speculations that user-initiated 
services provide better service to users than mediated 
ILL. Overall, user-initiated services have lower unit 



costs, higher fill rates, and faster turnaround times than 
mediated ILL. 

Mediated borrowing has improved compared to the 
1996 study. The current study reports a unit cost for 
mediated borrowing of $17.50 in 2002 dollars. When 
adjusted for inflation, that unit cost would represent 
$14.88 in 1996 dollars, compared with $18.35 in the 
previous study and represents a 19% reduction in 
constant dollars. Staff costs now represent less than 
60% of the unit cost. Photocopy as a percentage of the 
overall lending unit cost has decreased and borrowing 
fees now represent a larger percentage of the borrowing 
unit cost. 

Turnarormd time for mediated borrowing has 
decreased as has the volume of mediated borrowing and 
lending compared with the 1996 study. 

Bruce Kingma, Associate Dean, School of Information 
Studies, Syracuse University, and Tom Delaney, Head of ILL 
at Colorado State University, are collaborating with Mary 
Jackson, ARL, in the conduct and analysis of this study. For 
more information, contact Mary Jackson <mary@arl.org>. 

1 ARL/RLG Interlibrary Loan Cost Study: A Joint Effort by the Association 
of Research Libraries and the Research Libraries Group (Washington, 
D.C.: ARL, 1993). 

2 Measuring the Performance of Interlibrary Loan Operations in North 
American Research & College Libraries (Washington, D.C.: ARL, 

1998). 
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Time and Cost Analysis: 

Staff Allocations Project 

by Dilys Morris, President, Chronos Group 

I n February 2004, Notre Dame and Vanderbilt 

Universities will begin studying how staff fime and 
cosfs are allocafed across fhe library. The fwo libraries 
have worked fogefher wifh fhe Chronos Group fo 
idenfify library-wide fime cenfers. They will use fhe 
Chronos Group's TCA DecisionBase mefhodology and 
soffware fo collecf and analyze dafa. Vanderbilf will fesf 
fhe fime cenfers during fhe fall of 2003. 

The goal of fhe sfudy is fo undersfand fime and cosfs 
af fhe macro, or cenfer, level. Time dafa will also be 
collecfed af a micro, or fask, level. Bofh libraries will 
have a mix of macro and micro dafa since some 
adminisfrafive areas of each library prefer fo collecf dafa 
af fhe micro level. Since fhe cenfers represenf library 
funcfions, if will be possible fo see how fime is 
disfribufed by funcfion across fhe libraries. Addifionally, 
because reporfs can be sorfed by any adminisfrafive unif, 
fhe sfudy will show all funcfions performed by a given 
unif, including fhose fhaf fall oufside fhe normal work of 
fhaf unif. The libraries will be able fo examine how fhe 
acfual fime spenf compares wifh fheir priorifies and 
expecfafions, and evaluafe whefher if is possible fo 
allocafe more fime fo higher priorifies. Finally, if will be 
possible fo compare fime allocafions (bofh acfual fime 
and proporfions) wifh fhose of a similar insfifufion. 

The libraries will code sfaff according fo fhree broad 
employmenf groups: exempf from fhe 40-hour 
workweek (e.g., professionals), nonexempf, and sfudenfs. 
Comparisons by fhese employmenf groups will be 
possible and each library will enfer finer classificafion 
fypes for more specific local reporfing. 

The TCA DecisionBase fime-cosf benefifs analysis 
mefhodology combines a longifudinal approach and fime 
sampling. Four sample weeks will be selecfed annually. 
For fhe remainder of FY 2004, fwo weeks only will be 
sampled: February 23-29 and May 10-16. To facilifafe 
comparisons befween fhe libraries, no holiday weeks are 
included in fhe sample. 

During a sample week, all sfaff fime will be fracked 
by fhe fime cenfers. The cenfers are divided info fwo 
groups: Product/ Service Centers and Overhead Centers (see 
below for a lisf of cenfers). An Excluded Center fracks fime 
fhaf is nof used in reporfs, such as leave wifhouf pay. 

The separation of Overhead Cenfers from Producf / 
Service Cenfers will allow fhe libraries fo gef a clear 
picfure of fheir overhead cosfs. Reporfs idenfify fhe 
overhead cosfs for each layer in fhe organization. 

Cosfs of each adminisfrafive group are spread across 
all unifs supervised. 

Cosfs for a given cenfer consisf of employee salary and 
fringe benefifs multiplied by fhe fime spenf in fhaf cenfer. 
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Library-Wide Time Centers 



Product/Service Centers 

• Acquisitions 

• Catalog Maintenance 

• Cataloging 

• Circulating Materials 

• Collection Development 
& Management 

• Conversion or Major 
Projects 

• Development & PR 

• Digital Collections 

• ILL & Document Delivery 

• Instruction 

• Preservation 

• Storage & Stacks 
Maintenance 

• User Assistance 

• Volume Preparation 



Overhead Centers 

• Administrative & 
Support Services 

• Automation & Systems 

• Paid Leave 



Excluded Center 

• Time Not Counted 



Employmenf dafa for each employee is enfered once, buf 
can be updafed for subsequenf sample weeks as needed. 
Each employee has a Position Code fhaf reflecfs fhe organi- 
zational sfrucfure. Individuals who work in multiple 
organizational unifs are assigned multiple Position Codes. 

To collecf fhe fime sampling dafa, every employee 
will manually complefe a fime sheef fhaf idenfifies fhe 
fime by cenfers for fhe entire week. If fasks wifhin 
cenfers are used, fhe fime spenf af each fask will be 
recorded. Time is recorded in 15-minufe incremenfs, 
and rounded fo fhe nearesf quarfer of an hour. Break 
fime is nof recorded; insfead, employees will spread 
non-working fime over fhe cenfers or fasks worked. 

TCA DecisionBase calculafes fhe hourly salary of 
each employee for every sample week. The hourly salary 
is multiplied by fask or cenfer fime fo arrive af cosfs for 
each employee. The soffware sums fime and cosfs. 
Oufpuf measures can be enfered for each sample week 
and are used fo defermine unif cosfs and times per fask 
or cenfer. These measures (e.g., fifles cafaloged, maferials 
circulafed, quesfions answered, efc.) can be enfered for 
any fask or cenfer. 

Reporfs give bofh fime and cosfs for cenfers and fasks, 
and can be generafed for any defined adminisfrafive unif. 
Addifionally, reporfs can be generafed for fhe fhree 
employee groups mentioned above. Weekly and 
projecfed annual reporfs are produced. 



— Copyrighf 2003 Dilys Morris 



For further information, please contact Dilys Morris at the 
Chronos Group <dmorris@iastate.edu>, Flo Wilson at 
Vanderbilt <flo.wilson@vanderbilt.edu>, or }o Bessler at 
Notre Dame <JoanneM.Bessler.l@nd.edu>. 
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Evaluating the Utilization of 
Facilities in ARE Libraries 

by Clifford H. Haka, Director of Libraries, and Jim Hensley, 
Facilities Manager, Michigan State University Libraries 

H ow can libraries tell if they are effectively utilizing 
their space? Is there a way to measure how we 
currently utilize the physical space in research 
libraries? collect data and compare it across libraries? and 
then assess the impact of various space allocation decisions 
on library activity such as building use and circulation? 

For example, two libraries might pursue radically 
different approaches to shelving materials, with one using 
traditional shelving while the other utilizes compact 
shelving. This would result in the allocation of space for 
shelving in one library being much higher than in the 
other library, and enable us to assess if this has an impact 
on use, such as the total number of circulations. Similar 
examples might include the extent to which libraries 
subscribe to electronic journals, thereby limiting the 
space required for storage of physical journal titles, 
or the degree of outsourcing, which would decrease 
the amount of space required for staff operations. 

No single measure, such as the examples suggested 
above, would allow libraries to respond to the overall 
question of the future role of physical library facilities. 
Such measures should, however, provide insight regard- 
ing those factors that encourage, or possibly discourage, 
the utilization and subsequent effectiveness of library 
facilities. 

While a review of space utilization along these lines 
might not fit the classic definition of a "performance 
measure," this topic was, in fact, identified as an area of 
interest during the initial ARL -sponsored New Measures 
Retreat held in Tucson in 1999. Could procedures be 
developed that might assist in the assessment of current 
space utilization within ARL libraries? No "ready-made 
tool" exists that would enable a library to review this 

■ matter objectively and expeditiously. However, many at 
the retreat felt that it would be interesting, and potentially 
useful, to conduct a detailed review of space utilization 
within several ARL libraries to determine if trends, or 
possibly of more interest, abnormalities, exist in regard to 
the manner in which ARL libraries use their space. 

Library Space Evaluation at 
Michigan State University 

To get this ball rolling, the Michigan State University 
(MSU) Libraries undertook to establish detailed CAD 



files for the Main Library facility. This was done by 
employing CAD 2000 software. 

CAD files containing the basic layout of the library 
building were obtained from the Physical Plant 
Engineering Department. Unfortunately, these files proved 
to be incomplete and in some cases inaccurate. Therefore it 
was necessary to verify dimensions by re-measuring many 
areas before proceeding. Following the establishment of an 
accurate interior, additional elements such as stacks, service 
points, and furniture were added to the drawings. (As an 
aside, once completed these documents have proven to be 
extremely useful for space-plarming projects, especially 
when units move to new locations. Precise plans can be 
developed, reviewed, and modified without the tedious 
and time-consuming sketching and re-sketching, often 
inaccurately, that was the previous norm.) 

The next step was to define different types of spaces 
found within the typical library. Two primary resources 
were used, starting with Metcalf.^ Using guidelines 
contained within Metcalf, assignable and non-assignable 
space was identified throughout the building. The 
identification of non-assignable space — such as mechanical, 
custodial, and telecommunication areas — is an important 
first step as these spaces cannot typically be used, but they 
still make up a significant percentage of the overall square 
footage of all buildings. The next step was to discern what 
functions had been assigned to the remaining space. To 
help with these definitions, a second resource, published by 
the Buildings and Equipment Section of LAMA,^ was 
utilized, resulting in the following seven categories: 

• Non-Assignable: Areas designated for other uses — 
structural elements, HVAC chases, electrical vaults, 
telecommunication closets, custodial areas, etc. 

• Service: All areas that are points of contact between 
staff and patrons, or are areas for patron self-service 
such as copy machines and telephones. 

• Staff Area: Office space, work areas, and facilities- 
support areas such as loading dock and mailroom. 

• Stacks: All areas designated for storage of books 
and materials regardless of format. For instance, 
microfilm cabinets fall within this category. 

• Seating: All seating without network capability. 

• Networked Seating: All other seating that has 
network capability. 

• Traffic: These areas are used primarily for 
movement around the building and for no other 
purpose. Main corridors and hallways to elevators, 
service areas, and restrooms were included. 

It would certainly be possible to divide many if not all 
of these categories into countless subdivisions. However, 
every division requires further definition and increases the 
complexity of compiling the data for a library. Therefore, 
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in the hope that other ARL libraries might be willing to 
replicate these efforts to provide comparative data, we 
strove for a limifed number of sfraighfforward and 
unambiguous cafegories. 

Seafing serves fo depicf fhis dilemma. Differenfiafing 
befween carrel seafing, fable seafing, and "casual" seafing 
is nof fhaf difficulf. When one adds fhe elemenf of 
"laptop-ready seafing" however, which cufs across all 
fhese seafing fypes, if was decided, af leasf for fhe 
purposes of coding fhe MSU floor plans, fo collapse 
seafing fypes info nefworked and all ofher seafing. If was 
subsequenfly decided fo depicf addifional defail via ofher 
techniques. For example, a pie charf was developed fo 
display a more defailed range of seafing opfions, as well 
as fheir relative availabilify (see accompanying charf). 

To use such defailed distinctions when coding floor plans 
would, however, yield an indecipherable maze fhaf 
would be exceptionally daunting and time consuming 
fo mainfain wifh any degree of accuracy. Hence, we 
pursued fewer cafegories for fhe overall coding of fhe 
floor plans wifh defail represented via alfernative charf s. 

If would seem fhaf delineafing and marking fhe plans 
info fhese seven cafegories would be relatively 
sfraighfforward af fhis poinf. This proved nof fo be fhe 
case. This parf of fhe projecf was more fime-consuming 
fhan anficipafed. For example, defining spaces for 
seafing fhaf are adjacenf fo walkways is difficulf fo 
measure in fhaf fhe chair push-back space from fhe fable 
or carrel frequenfly encroaches info fhe area defined as a 
walkway. Specifically fhe push-back from a 30-inch work 
surface carrel was defined in fhe LAMA documenf as 4 ff. 
6 in., a significanf amounf of floor space fhaf frequently 
spilled into walkway space. In a very real sense this was 
the confirmation that original space allowances had been 
discarded, at least in this instance at the MSU Library. 
This space was then defined as dual purpose and if 
became an issue fo address. In fhis parficular case if was 
decided fhaf fhe hallway space took precedence and was 
measured as such. However, fhere were many such 
measuremenf decisions fhaf made "fine funing" of our 
CAD documenf somewhaf challenging. Lisfing all of fhe 
various definifions and/or conventions we considered 
too exhausfive for fhe purposes of fhis paper. However, 
mosf are found in fhe fwo sources cifed. 

Upon completion of floor plan coding,^ if was 
relafively sfraighfforward fo produce charfs and graphs 
fo depicf specific aspecfs of building use. In furn, fhis 
enabled fhe calculafion of average space allocations for 
items such as seafing. For example, one calculafion made 
across all fypes of seafing found fhaf an average of 31 
square feef per seaf has been provided, which falls well 
wifhin fhe sfandard esfablished by Mefcalf . Information 
such as fhis can prove useful in considering issues such 
as, "Have foo many seals been infroduced info foo small 



MSU Space Utilization, 
Seating by Type 



Seating /special format (microfilm) 0.44% 



Seating /group 
42% 



Seating /computer 
10% 




Seating /carrel / 
netlink 



Seating /fac / 
grad /carrel / 
netlink 
3% 



Seating/ 

a/v 

2% 



Seating /carrel 
25% 



Seating /casual 
9% 



Charf created by Jin-hee Kim, MSU Graduate Student 



of a space?" Calculafions for book-sfack capacify would 
be similarly insfrucfive and pofentially useful in 
argumenfafion for addifional space. 

The creation of fhe CAD drawings was fhe mosf 
complex and fime-consuming porfion of fhis projecf. 
However, the hours spent on this task have paid huge 
dividends for space-planning discussions. The process 
is, of course, ongoing as sfaff and users move and 
relocafe chairs, seafing, and ofher items. 

The MSU Libraries plan fo updafe and review fhese 
documenfs on a regular basis. We hope fhaf we can 
engage one or more ARL libraries fo become parfners 
with us to further define fhis process. Our goal is fo 
creafe a fool fhaf will prove helpful for comparafive 
purposes and fo assess fhe impacf of various space 
allocation decisions on library acfivify. If a group of 
ARL libraries can be idenfified fo compile comparable 
dafa, fhis could provide "benchmark measures" againsf 
which ofher libraries could compare fheir sifuafions. 

We look forward fo fhis possibilify. 

— Copyrighf 2003 Clifford H. Haka and Jim Hensley 



1 Keyes D. Metcalfs Planning Academic and Research Library Buildings 
(New York, McGraw-Hill: [1965]) has been employed in countless 
library construction projects over the past 35 years. 

2 American Library Association, Library Administration & 
Management Association, Buildings and Equipment Section, 
Functional Space Requirements Committee, Building Blocks for 
Planning Functional Library Space (Lanham, Md.: Scarecrow Press, 
2001 ). 

^ MSU's coded floor plan and samples charts and graphs produced 
with the CAD software are available on the ARL Web site 
<http: / / www.arl.org / newsltr / 230 / space.html>. 
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Creating a Culture oe 
Assessment: The University oe 
Arizona Experience 

by Carla Stojfle, Director of Libraries, and Shelley Phipps, 
Assistant Dean for Team and Organizational Development, 
University of Arizona Library 

Especially given her leadership role in launching the 
ARL New Measures Initiative, it is not surprising that 
Carla Stoffle has led the University of Arizona Library to be 
an early adapter of performance measures. The article below 
describes how one research library is assessing itself by 
implementing a performance measurement system and 
integrating the results of the New Measures Initiative 
projects as they become available. The benchmarks identified 
and described by Arizona are made public to provide other 
libraries with a point of departure for measuring their own 
operations. This article is reprinted with permission from 
the authors and the Editor 0 / Library Issues, where it 
appeared in the March 2003 issue, vol. 23, no. 4. 

I n addition to the national efforts that have set 
a new direction for performance measures in 
academic research libraries, individual insfifufions 
are experimenfing wifh using assessmenf fechniques 
fo quickly learn abouf changes in local users' 
expecfafions. The imporfance of implementing an 
organizational performance measuremenf sysfem has 
been one conclusion of such experimenfs. One 
insfifufion, fhe Universify of Arizona Library, creafed 
a Performance Effecfiveness Managemenf Sysfem 
(PEMS) in 1998. 

PEMS is designed fo align individual, unif, and 
whole organization efforfs wifh sfrafegic goals sef by 
fhe Library and fo provide measures fhaf indicafe 
success, progress, and fhe need for improvemenf. 
Developing a sysfem approach fo measuremenf helps 
develop an infernal culfure of assessmenf where 
decisions are guided by facfs, research and analysis, and 

■ where services are planned and delivered fo maximize 
positive oufcomes for customers and sfakeholders. 

Sfaff are encouraged: 

• fo care abouf whaf resulfs fhey produce; 

• fo value fhe acfual impacf fhey have on fhe 
educational or research process; and 

• fo know how fhese resulfs relafe fo user 
expecfafions. 



Creafing fhis culfure can ensure fhaf fhere is a 
commifmenf fo confinuing assessmenf, developmenf of 
new fools, and use of dafa for confinuous improvemenf. 
Wifhouf an emphasis on culfure change, barriers will 
remain fo fhe full accepfance of fhe use of performance 
measures. 

PEMS: How Does It Work? 

The PEMS system requires units to continuously assess 
needs of fheir assigned faculfy and sfudenf groups. 
Using varying fechniques, feams engage in discovering 
whaf is mosf imporfanf fo fheir customer groups abouf 
fhe services fhey provide. They use fhis information fo 
formulate sfandards or performance fargefs for fhe 
parficular service acfivify. They gafher dafa on progress 
toward fhese sfandards and reporf periodically fo fhe 
Library Cabinef and Sfrafegic Planning Team. 

Affer analyzing fhe needs and assessing where 
improvemenf can make a difference, fhe feams eifher 
engage in specific projecfs fo increase fheir effecfiveness 
or assign individuals fhe responsibilify for increasing 
fhe amounf or qualify of fheir individual work fhaf 
resulfs in oufcomes for sfudenfs and faculfy. Thus fhe 
sysfem supporfs individual goal seffing fhaf furfhers fhe 
unif's service effecfiveness related fo whaf cusfomers 
have idenfified as imporfanf. Sfrafegic, fufure focus is 
sfrucfured info fhe sysfem as fhe feams sfarf fhe year's 
planning by undersfanding fheir role in achieving fhe 
Library's fhree fo five-year sfrafegic goals. 

As feams creafe fheir sfrafegic framework for fhe 
year, fhey sef oufcome, oufpuf and qualify measures fo 
assess success. In some cases oufcomes are measured 
direcfly wifh cusfomers. In ofher cases, needs 
assessmenfs lead fo fhe creafion of oufpuf or qualify 
measures fhaf can be deduced from fhe idenfified need. 
For example, if a feam assesses fhaf only a limifed 
portion of ifs pofenfial user group is using ifs services, 
fhey may sef a sfandard fo increase fhe number of 
insfrucfional sessions or increase fhe number of 
sfudenfs reached. If fhey recognize fhaf limifed budgefs 
require selection of materials fhaf have a high pofenfial 
for usage, fhey may sef a sfandard fhaf measures acfual 
usage of material purchased. If fhey idenfify fhaf 
timeliness is an imporfanf aspecf of fhe service fhey 
give, fhey may sef a sfandard for furnaround fime, from 
requesf fo delivery. If fhey idenfify fhaf accuracy of 
information is crifical fo fhe users of fheir service fhey 
may sef a relafed sfandard. 

Examples. Examples of oufcome measures may 
include: 95 percenf of fhe serials acquired in 
science /engineering areas are used af leasf once each 
year. 

Some sfandards use increased oufpuf as surrogate 
measures for an infended oufcome. For example, 95 
percenf of UA faculfy will receive intormafion abouf 
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intellectual property rights and copyright (the intended 
outcome is that faculty are able to use this information 
in their teaching and publication efforts). 

Some standards relate to the user's need for access 
or service in a fimely fashion or a need for accuracy fhaf 
will facilifafe access: 90 percenf of fradifional reserve 
requesfs af all sifes will be available fo cusfomers wifhin 
48 hours affer receiving fhe requesf. 

Assessmenf wifh users has led fo process 
improvemenf efforfs fhaf have decreased cycle time, 
increased amounf and qualify of service, and saved 
hundreds of fhousands of dollars fhaf have been 
reallocafed fo fhe purchase and 
refresh of fechnology and fhe 
implemenfafion of a compefifive 
salary sfrucfure. 

Learning fhe fools and 
mefhods for assessing 
performance has been a 
challenge. The ARL New 
Measures Initiative has afforded fhe opporfunify fo use 
more reliable measuremenf mefhods fhaf will also yield 
peer benchmarking dafa. The Library's Sfrafegic Plan 
now incorporafes measures fhaf can be derived from fhe 
LibQUAL+T*^ insfrumenf as well as measures based on 
feam dafa gafhering. For example: 

• Degree fo which access fo elecfronic information 
exceeds cusfomers' minimum expecfafions (from 
LitiQUAL+T*^): 15 percenf increase wifhin 5 
years. 

• Scholarly elecfronic resources newly developed 
by fhe Library and available remofely fo 
cusfomers (120 new resources wifhin fhe nexf 5 
years). 

Using Data for Strategic Decision Making 

Information from fhe firsf pilof years of LibQUAL+^’“ 
indicafed fhaf users desired delivery of elecfronic 
information fo fhe deskfop and fhaf fhe Library was nof 
reaching even minimum expecfafions. A sfrafegic 
decision was made fo creafe a Documenf Delivery Team 
and sef sfandards for elecfronic delivery of inferlibrary- 
loaned maferial and reserve arficles. Cosf information 
from fhe ILL/DD projecf was used fo research besf 
practices and fo join wifh ofher libraries using more 
efficienf fechnology. This enabled cusfomers fo order 
inferlibrary loans direcfly from ofher libraries, which 
reduced furnaround time. A sysfems analysis projecf 
feam was formed fo research fhe capabilify of presenf 
sysfems fo provide fhe infrasfrucfure necessary fo 
improve access in fhe fufure. As a resulf, fhe UA 
Library is now parficipating in fhe developmenf of fhe 
"Scholar's Porfal" soffware wifh a number of ARL 
Library parfners. 



Access fo elecfronic informafion is indicafed as highly 
desired by LibQUAL+T*** respondenfs and confirmed by 
feam inferviews and surveys. This informafion also 
confribufed fo fhe seffing of fhe sfandards in fhe Sfrafegic 
Long Range Plan and influenced fhe UA Library's 
decision fo invesf a larger portion of ifs informafion 
access budgef in elecfronic informafion resources. 

Evaluation of fhe Universify's budgefary sifuafion 
has led fo plans fo increase consorfial purchases as a way 
of reducing overall cosfs fo fhe Universify and a sfandard 
was sef fo save af leasf $100,000 per year on such 
purchases. The UA Library will be joining fhe SAILS 

projecf fo research more efficienf 
ways of measuring learning 
oufcomes. As fhe Library builds 
ifs new fechnological 
infrasfrucfure, we will be using 
fhe research from fhe E-Mefrics 
projecf fo assess fhe 
effecfiveness of our elecfronic 
access sysfems — from fhe cusfomer poinf of view and in 
comparison wifh peers involved in fhis nafional efforf. 

Fulfilling our Educational Mission 

The PEMS system and its ability to use the results of 
national initiatives such as the "New Measures 
Initiatives," enables the Library to demonstrate to the 
campus that all resources and staff efforts will be focused 
on the changing expectations of our users by providing 
value-added services. Incorporating efficiency measures 
and developing process improvement projects also 
demonstrates the Library's commitment to maximizing 
return on investment. Accountability is demonstrated at 
the institution, unit and individual level. 

Use of performance measurement is a way of 
ensuring that academic libraries retain their ability to 
perform their special mission within the educational 
process. These include: 

• Valuing freedom of access to information, 

• Eliminating censorship, 

• Providing equitable access to all levels of users, 
and 

• Increasing the information literacy of the students 
and faculty in this increasingly complex scholarly 
communication system. 

Developing systems and using evolving tools created 
by ARL-sponsored projects and others, increases the 
ability to measure and communicate performance 
outcomes and the effect of libraries on the quality of 
teaching and learning processes in our institutions. 

— Copyright 2003 Mountainside Publishing Co., Inc. 



. . . teams engage in discovering what is 
most important to their customer groups 
about the services they provide. 
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Using Data to Make Choices: 

The Balanced Scorecard at the 
University oe Virginia Library 

by Jim Self, Director of Management Information Services, 
University of Virginia Library 

I n early 2001 the University of Virginia (U.Va.) Library 
began implementation of fhe Balanced Scorecard. 

This inifiafive confinues a long hisfory of collecting 
and using dafa fo improve services, collecfions, and 
processes af fhe U.Va. Library. 

The Balanced Scorecard (BSC) allows an 
organization fo focus on a relafively small number of 
carefully chosen measuremenfs. These measures are 
aligned wifh fhe library mission and sfrafegies, and 
fhey provide a quick buf comprehensive picfure of 
organizafional performance. 

The concepf dafes from a 1992 article in fhe Harvard 
Business Review by Kaplan and Norton.^ The scorecard 
was designed for privafe business, buf is increasingly 
being used by governmenfal and nonprofif organizations. 
In 2001 ARL and OCLC sponsored a fhree-day forum on 
performance measures, which feafured fhe Balanced 
Scorecard.^ Several German libraries, imder fhe leadership 
of fhe German Research Council, have implemenfed fhe 
Balanced Scorecard.^ However, implemenfafion in Norfh 
American libraries has been slow fo develop. 

The Balanced Scorecard examines fhe organization 
from four perspecfives: User, Finance, Infernal 
Processes, and Learning and fhe Fufure. Four fo eighf 
measuremenfs, or mefrics, are devised for each cafegory 
or perspecfive. Each mefric has a specific and 
unambiguous fargef or sef of fargefs. For example, fwo 
of fhe mefrics adopfed by U.Va. for 2002-03 are 
described below. Each of fhese mefrics confains fwo 
fargefs. Targefl indicafes complefe success in achieving 
fhe mefric; Targef2 indicafes parfial success. 

To recruif, develop, and refain productive, highly 
qualified sfaff, fhe library esfablished a mefric for fhe 
Retention Rate of Commended Employees: 

Targetl: 95% of commended employees remain on 
fhe sfaff. 

Target2: 90% of commended employees remain on 
fhe sfaff. 

Method: Staff members who have received af leasf 
one exfraordinary confribufor wrife-up and faculfy 
who have received af leasf one "exfra-meriforious" 
or "highesf" salary increase over fhe pasf fhree 
performance cycles are identified af fhe beginning of 
each fiscal year. The library may also fake ofher 
forms of recognition info considerafion. Af fhe end 
of fhe year, we will defermine whaf percenfage of 
fhese employees still work af fhe library. 



To provide resources and services fhaf have a high 
rafio of value fo cosf, fhe library has esfablished a mefric 
for Expenditures for Circulating Monographs as a Proportion 
of All Monographic Expenditures: 

Targetl: Monographs that circulate within the first 
two years should account for af leasf 60% of fhe 
monographic budgef. 

Target2: The circulafed ifems should accounf for af 
leasf 50% of fhe monographic budgef. 

Method: A yearly study similar to the 1993-95 
baseline study will be conducted. The study will 
track purchased monographs for fwo years fo 
defermine if a customer uses fhem. 

The U.Va. Library is using fhis fechnique for several 
reasons. In essence fhe Balanced Scorecard enables us 
fo gain beffer confrol of our sfafisfical operations. By 
limiting fhe number of scorecard mefrics, if forces us fo 
decide whaf is imporfanf, and fo idenfify fhose numbers 
fhaf fruly make a difference. If also infroduces some 
balance info our sfafisfical work. Like many libraries, 
we have collected many dafa regarding resources and 
user services, buf ofher areas have nof received fhe same 
attention. The BSC compels us fo fake a focused took af 
finance, infernal processes, and fhe fufure. Anofher 
imporfanf aspecf of fhe BSC is fhe assigning of fargefs or 
goals. We nof only decide whaf measures are 
imporfanf; we also sfafe whaf consfifufes success for 
each measuremenf. A final rationale for fhe scorecard is 
ifs infelligibilify; each year we can liferally see how well 
we have done. We gef a clear picfure of organizafional 
performance in fhe form of a sef of pie charfs. We have 
one pie charf for fhe overall resulfs, and individual 
charfs for each of fhe four perspecfives. We can 
evaluafe, af a glance, fhe performance of fhe library. 

If is equally easy fo compare fhe four perspecfives 
and fo nofe where we are doing well, and where we 
need fo improve. 

Choosing fhe mefrics may be fhe mosf crucial parf 
of fhe process. Once a mefric is esfablished, if has 
organizafional approval and recognition — as fhe saying 
goes: whaf gefs measured, gefs managed. The choice 
of mefrics depends upon fhe values of our organizafion. 
There musf be some agreemenf as fo whaf maffers, whaf 
counfs wifhin fhe library. Adopting fhe Balanced 
Scorecard requires fhe library fo confronf issues in a 
more rigorous fashion, fo make some hard decisions, 
and fo make fhose decisions explicif. 

The subjecf maffer of fhe mefric (e.g., reference 
service, cafaloging, fundraising, inferlibrary loan) is 
obviously imporfanf, buf nof fhe only issue. The nafure 
of fhe measuremenf is equally imporfanf. Do we counf 
fhe number of fimes we perform a fask? Do we measure 
fhe cosf of performing fhe fask? Do we calculate fhe 
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The Scorecard Measures are "Balanced" into Four Areas 



User Perspective 

How can we fulfill user 
expecfations? 

• Satisfaction rafing of our customers 
in our periodic surveys 

• E-mefrics 

• Turnaround fime for pafron requesfs 


Finance Perspective 

How can we secure and use resources 
in an effective way? 

• Unit cost for various fransacfions 

• Cosf per use of purchased maferials 

• Revenue from sfafe and privafe sources 


Internal Process Perspective 

How should we organize infernal processes 
fo meef user needs and expecfafions? 

• Throughpuf fimes for making 
available new materials 

• Turnaround times for internal 
service requests 


Future Perspective 

How can we secure the future capability 
of our services? 

• Sfaff fraining oufcomes 

• Employing fhe besf people 

• Replacemenf / updating of equipmenf 
and fechnical resources 



time it typically takes to do the task? Or do we survey 
our customers and ask them how well we performed 
the task? Any of fhese fechniques mighf be appropriafe, 
buf our choice of fechniques is reflecfive of fhe library's 
priorities. 

The mefrics also reflecf practical considerations. 
Collecting dafa generafes cosfs, and we affempf fo 
minimize fhose cosfs. We do nof wanf fhe Balanced 
Scorecard fo be a burden; if should nof appreciably 
increase our cosfs — eifher in ferms of labor or monefary 
expendifures. Whenever possible, we use existing 
measuremenfs, incorporating fhem info fhe process as 
scorecard mefrics. The second choice is fo use dafa fhaf 
can be mined efficientiy from dafabases. The fhird 
choice is collecfion fhrough sampling — preferably 
carried ouf by sfudenf employees. Only as a lasf resorf, 
do we ask operational ("fronf line") sfaff fo spend fheir 
own time collecting dafa. 

U.Va. has collecfed scorecard dafa for fwo fiscal 
years. We are currenfly completing fhe FY03 fally, and 
will soon communicafe fhe resulfs fo library managers 
and sfaff. Af fhe same fime we are collecting dafa for 
FY04, and sfarting fhe planning for FY05. 

The process is sfill evolving, buf fhus far we are 
very pleased wifh fhe Balanced Scorecard as a 
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managemenf fool. If has improved our sfatisfical focus, 
clarified and balanced our organizational values, and 
helped us develop a culfure of assessmenf. We fhink ifs 
ufilify will only increase as we become more 
experienced in ifs implemenfafion. 

— Copyrighf 2003 Jim Self 

Additional information can be found at <http://www.lib. 
virginia.edu/bsc/> or by writing to <self@virginia.edu>. 

IR.S. Kaplan and D.P. Norton, "The Balanced Scorecard — Measures 
that Drive Performance," Harvard Business Review 70, no. 1 (1992): 
71-79. 

^ARL/OCLC Strategic Issues Forum, "Toward Rethinking Academic 
Library Performance in the Digital Age," Tempe, Arizona, February 
16-18, 2001, <http:/ / www.arl.org/olms/forum2001.html>. 

3 R. Poll, "Performance, Processes, and Costs: Managing Service 
Quality with the Balanced Scorecard," Library Trends 49 (2001): 
709-717. 
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Data-Informed Decision Making 

hy Susan }. Beck, Head of Public Services, Paul Robeson 
Library, Rutgers, the State University of New Jersey 
As part of her sabbatical, Susan Beck served as an 
ARL Visiting Program Officer in 2002 to investigate the 
impact of assessment on library management decision making 
and the degree to which assessment data has influenced 
change. She presented papers on the results of her research 
at the ACRL 11th National Conference and the 5th 
Northumbria International Conference on Performance 
Measurement in Libraries.^ 

I n this age of increased accountability, academic 
research libraries must effectively present 
information on their accomplishments to 
university administrators. Often, these administrators 
are looking for hard data as evidence of organizational 
improvement. In order to learn more about how data 
can influence decisions, nine ARL public universities 
actively involved with assessment activities were 
examined for patterns in their use of data to make 
decisions (Arizona, Arizona State, Connecticut, Texas, 
Toronto, Virginia, Virginia Tech, Washington, York). 

Fifty -nine senior library managers and directors 
were interviewed about issues such as accountability, 
governance, service evaluation, plarming, decision 
processes, assessment tools, and new data measures. 
Interview responses were analyzed using Atlas. ti: The 
Knowledge Workbench, a qualitative data analysis tool. 

As expected, library administrators use data for 
planning and setting organizational priorities. Data are 
necessary for developing policies, allocating staff, 
building collections, and planning service delivery. The 
kinds of data used include formal assessments, survey 
results, and anecdotal information. Informed decision 
making lends credibility to both the library and its 
leadership. At some institutions the libraries have taken 
the lead in providing data, while at others the parent 
institution requires that all funding requests be 
accompanied by appropriate data. One library has a 
formal Management Information System by which it 
can generate the data necessary for units to support 
their planning processes. Some library administrators 
also feel that it is important to encourage data-driven 
decisions at all levels in the organization. 

Nearly every unit in the library can use data for 
decision making. Collection development data for both 
print and electronic resources include usage data, 
survey data, economic information, departmental needs, 
shelving and interlibrary loan statistics, server logs, and 
information from faculty consultations. Some librarians 
have developed specialized local tools such as databases 
or Web sites. Special and digital collections data 
include information about users, research topics, and 
materials use. Usability testing of digital resources and 



server log analyses can help determine how those 
collections can best be used. 

Human resources can conduct climate surveys to 
determine work satisfaction and identify problem areas. 
They often use data from focus groups, user (staff) 
surveys, and exit interviews. Analyses of peak service 
use can highlight staffing patterns to provide informa- 
tion for better scheduling. Process improvement studies 
can be used to discover areas for change in allocating 
staff resources. 

Data are critical for making decisions about 
library space and equipment. Increased demand for 
instructional space points to the need for additional 
classrooms and sophisticated equipment. Performance 
of equipment can be tracked through logs and repair 
records. Student surveys often provide the needed data 
to support proposals for increased technology. 

Decisions about services are influenced by user 
surveys, use data, focus groups, budget cuts, and 
process improvement activities. As an example, library 
service hours in all libraries visited changed as a result 
of data on use of services. Other services in which 
changes were made included interlibrary loan, desktop 
delivery and virtual reference services, shelving, and 
electronic reserves. 

Results of the interviews indicate that assessment 
data provide necessary evidence to implement change 
in research libraries. Each visited library could point to 
several examples of the successful use of assessment 
data. Libraries that create, nurture, and integrate 
assessment into their everyday activities are models for 
other institutions. The willingness of leadership to 
incorporate assessment data into decision-making 
activities can be reflected in their own interest and the 
organizational culture, as demonstrated in staff 
expertise and concern for service quality. 

Accountability pressures from the parent institution are 
also a significant factor for these institutions and the use 
of assessment data provides the library with an 
opportunity to highlight its accomplishments with 
credibility and integrity. 

— Copyright 2003 Susan J. Beck 

1 Susan J. Beck, "Making Informed Decisions: The Implications of 
Assessment," paper presented at ACRL 11th National Conference, 
Charlotte, N.C., April 10-13, 2003; and "The Extent and Variety of 
Data-lnformed Decisions at Academic Libraries: An Analysis of 
Nine Libraries of the Association of Research Libraries in the U.S.A. 
&c Canada," paper presented at 5th Northumbria International 
Conference on Performance Measurement in Libraries, 

ColUngwood College, Durham, England, July 28-31, 2003. 
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ARL Service Quality 
Evaluation Academy 

by Consuella Askew, LWQUAL+™ Program Specialist 

O n May 16, 2003, ARL's Service Quality 

Evaluation Academy graduated its second class 
of participants at the close of an infensive five-day 
workshop in San Antonio, Texas. The Academy 
emphasizes basic concepfs and skills in measuremenf and 
dafa analysis for library service qualify evaluation. The 
curriculum focuses on qualifafive and quanfifafive 
mefhods wifh some fime spenf on relevanf dafa-analysis 
soffware, such as ATLAS.fi and SPSS. The workshop is 
directed towards conducfing service qualify evaluations 
in general, wifh fhe expecfation fhaf parficipanfs will 
experimenf wifh mefhods and evenfually create fheir own 
fools to enhance fhe ones being developed by libraries. 

The Curriculum 

The firsf half of fhe workshop was devofed to 
quanfifafive mefhods and was led by Bruce Thompson, 
Professor and Distinguished Research Scholar from fhe 
Deparfmenf of Educafional Psychology af Texas A&M 
Universify. Parficipanfs benefifed from class lecfures as 
well as hands-on experience using SPSS soffware. 
Thompson infroduced sfafistical ferminology, rules for 
sfafisfical analysis, and sfafistical mefhodologies fo 
defermine validify, reliabilify, and significance. 

The second half of fhe workshop, led by Colleen 
Cook, Inferim Dean of Universify Libraries, Texas A&M 
Universify, concenfrafed on qualifafive research 
mefhodology. This segmenf of fhe program began wifh a 
half-day lecfure on fhe use of qualifafive mefhodologies. 
Then parficipanfs were infroduced fo focus group 
inferview techniques. Parficipanfs were assigned fo 
small groups and provided wifh inferview franscripfs, 
which fhey coded using ATLAS.fi and fhen discussed 
wifh fhe enfire class. 

Participant Feedback 

Participants' reaction to the Academy was overwhelmingly 
positive, with many wishing they could devote more 
than one week to the event. 

The ultimate goal of the Academy is to develop a 
critical mass of individuals with assessment skills within 
the library profession. A majority of the participants 
recommended that a follow-up training session would 
facilitate their use of the tools and methodologies taught in 
the Academy. They also indicated that they would enroll 
in an online course that enhanced what they learned in this 
workshop. As we listen to our participants, it is evident 
that, while the Academy is progressing toward its goal, 
there are additional learning opportunities to explore. 

For information about participating in the next ARL 
Service Quality Evaluation Academy, contact Consuella 
Askew <consuella@arl.org>. 
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Creating a Culture oe 
Assessment — An ARL /OEMS 
Workshop 

C ustomer- or user-centered decision making 
and programming requires relevant data and 
information but also an organizational bias toward 
continuous assessment. The ARL Office of Leadership 
and Management Services (OLMS) offers a workshop on 
Creating a Culture of Assessment by investigating the 
main forces influencing the library and how these forces 
beg for the adoption of assessment and evaluation as an 
integral part of library work. 

Workshop participants practice: integrating 
assessment work into organizational systems and 
structures and everyday work activities; determining 
current and preferred future paradigms and skills to 
move groups toward the new paradigm; and customer- 
focused data gathering. 

For more information, including workshop schedule 
and registration form, see <http:llwww.arl.orgltrainingl 
institutes! cidture.html>. 



Measuring Library Service 
Quality — An ARL /OLMS 
Online Lyceum Course 

M easuring Library Service Quality is a Web-based 
course , offered over a six-week period by the 
ARL/QLMS Qnline Lyceum. The course covers 
the background and theory of measuring service quality, 
methods to assess and improve service, and the impact of 
measuring service quality on overall library value to con- 
stituencies. This course is designed for those interested in 
improving library service quality through a better imder- 
standing of how services are utilized, perceived, and 
assessed by users. This Qnline Lyceum course incorporates 
elements of both synchronous (real-time) and asynchro- 
nous interaction with course facilitators and a global peer 
network of up to 30 learners via a course bulletin board, 
chat rooms, and regular e-mail. 

For detailed course information, including a course 
schedule, visit <http://www.arl.org/training/quality.html>. 



Staying Current with the ARL 
Statistics & Measurement Program 



T o keep up to date with ARL Statistics and 
Measurement program activities, visit 
<http://www.arl.org/stats/>. You may also 
subscribe to the ARL new measures electronic mailing 
list. Members of the list are welcome to share ideas and 
information about performance measures, quality 
initiatives, and characterization of research libraries for 
discussion. See <http://www.arl.org/stats/ 
newmeas / newmeas_list.html>. 
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